 All right. Thanks everyone for coming and watching this presentation. We will be talking about a self-serve feature engineering platform using Flight and Feast. So the agenda for this talk is we're going to start off with some background on feature engineering. Then we're going to talk about Feast, then Flight. Then we're going to talk about Feast and Flight in combination, how those two can be put together to do feature engineering more effectively. Then we're going to have a quick demo and finally a conclusion. So for a bit of background, I'm sure most folks here are probably relatively aware of what feature engineering refers to, but of course to give a quick recap, one of the biggest problems today in machine learning is how to use data properly. And in particular, transforming raw data into data that's actually usable by ML models is one of the keys to effectively being able to train machine learning models and put them into production. So we call this process of transforming raw data into usable features feature engineering. So there's a whole bunch of challenges around feature engineering, and we're going to talk about three of them right now in this introduction. The first challenge is that building feature pipelines is extremely difficult. Now, this isn't entirely unique to machine learning. Building data pipelines in general is already a very difficult task. But specifically, when it comes to machine learning, building these feature pipelines is famously a very time consuming activity in machine learning, but also very high ROI, so extremely important. And there's this famous engineering quote that people like to toss around, which is applied machine learning, is basically feature engineering. So this is really one of the most critical parts when it comes to doing applied machine learning. So why is it hard to build these feature pipelines? Well, first of all, each individual machine learning project often requires standing up an individual pipeline for that project. And so that already leads to a lot of these different kinds of pipeline projects. And moreover, if you look at an individual feature pipeline, it can often require a whole host of different technologies. For instance, sometimes you're using pandas, other times you might be using Spark, Beam, Flink, Ray, SQL queries, all sorts of different things. And on top of that, your data can be coming from all sorts of different data sources. Nowadays, you often have data lakes. You have data warehouses. You also have streaming sources for real-time use cases. And so when you put all of this together, it really starts adding up to a lot of complexity. Finally, if that wasn't enough, you often have performance requirements. And these performance requirements might not be very strict for obviously a training situation. But if you want to productionize a machine learning model, then this gets a lot more difficult. And the last thing I'll say is that it's not just about authoring these pipelines and writing them once. You oftentimes need to schedule jobs to make sure your data is actually fresh and up to date if you're going to be using these models in production. So this adds up to a lot of difficulty. The second problem we're going to be talking about is consistent data access. So oftentimes, there's redevelopment of these machine learning pipelines. And this happens because a data scientist first will create a pipeline in order to get the features to train their model. And this can be represented in this diagram as the black flow on top labeled A, which shows a training pipeline. But by the time the data scientist is done training this model and wants to put it into production, often times this requires a data engineer or an ML engineer to step in and actually productionize this model, which typically leads to a whole new pipeline. And this is, of course, because the data scientist didn't build this pipeline in a very scalable, efficient, reliable, bug-free way. And of course, the ML engineer has to step in and do all this work. So you often end up with two different pipelines. And of course, this can oftentimes lead to inconsistency if you're not careful. If the data engineer doesn't exactly copy what the data scientist did, you might have some training serving skewed. And of course, this is going to really degrade your model performance. Because the model was trained on one pipeline, but it's running inference on a different pipeline. And of course, that's going to really harm your performance. And even worse, this isn't really that easy to detect because no one's going back and looking at the pipeline that the data scientist built. So this is also a really big problem when it comes to feature pipelines and feature engineering. The third challenge we're going to talk about is duplication of effort. So teams are often redeveloping these feature engineering pipelines. And why would this happen? Why isn't there collaboration? That's really because there's not an easy way for teams to collaborate today. How is team A supposed to know that team B is working on these features to support that model? And maybe there's some overlap. There's no really easy way to do this today, especially if your organization is super large and, say, at the scale of hundreds of data scientists and ML engineers, it's often very difficult to know what all the other teams are working on. So this can lead to pipelines getting rewritten from scratch. Moreover, even if you are aware of other teams who are using similar features, oftentimes you don't really want to trust them. Oftentimes, these machine learning models are very important for the bottom line of your company, for perhaps your team's success. And you might not actually be willing to put a lot of trust or faith into a pipeline that's been authored by some other ML engineers or by some other data scientists. So this leads to a lot of silo development, lack of collaboration between teams. Now, to be fair, a lot of these problems are less important in smaller organizations. But as you do scale, these problems get a lot more serious. So those are a bunch of the issues when it comes to feature engineering. And so now we're going to move into talking about Feast. Feast is an open source feature store. And what is a feature store? Well, a feature store is really an operational data system that's meant to solve all of the feature engineering problems that we talked about earlier. It's really meant to specialize in making it easy to create and put your ML features to use for training and production use cases for machine learning. Feast, as I mentioned, is an open source feature store. We entered the Linux Foundation for AI in September of 2020. So that was about a year ago. We have a very active community with contributions from pretty large tech companies, such as Robinhood, Salesforce, Gojek, IBM, Tecton, Redis, and Microsoft. We also have a lot of relatively smaller companies now who are trying out Feast, contributing back to the project, and generally trying to integrate Feast into their ML workflows. We have bi-weekly community calls, and our slack is pretty close to reaching 3,000 members. So if you want to, please stop by and say hello. So Feast is really focused on operationalizing data for ML on the modern data stack. So we lean very heavily into the cloud-native and serverless experience. We have very good support for AWS and GCP. We also have a really strong focus on being modular and extensible. So Feast, as I said, is really focused on solving these core feature engineering problems. It's not really meant to be a generic database. It's not meant to do data quality monitoring, like a tool like Great Expectations might do. It's certainly not meant to do anything related to training or more upstream, I guess, downstream tasks in machine learning. So it's really meant to solve these core operational data problems specifically for machine learning. In terms of the modularity and extensibility, Feast comes with pretty sensible defaults. So it's extremely easy to just run it immediately. You pip install it. You hook it up to a database or two, and then it's pretty easy to start exploring with. So we would definitely encourage you to try that. For folks for whom these defaults aren't enough, we made a lot of the core abstractions really extensible. So a lot of the bigger companies that I referenced earlier, they don't find that our immediate default implementations are sufficient enough, say, for their scale or something like that. And so a lot of them have written their own custom components and plugged them into the general Feast infrastructure and architecture. And they're able to use Feast in that way. As a specific example, a lot of people don't want to use one of our two main offline stores, which are BigQuery for GCP folks and Redshift for AWS. So now there's a bunch of community plugins for different offline stores. There's Snowflake, there's Hive, there's Signups on Azure, and there's even some folks building some offline stores on Alibaba Cloud. So a lot of different plugins. So now that we've talked about Feast, why don't we go back and talk about how Feast can help solve the feature engineering problems we were talking about earlier? So the first problem was that building feature pipelines is hard. How does Feast help with this? Well, if you look at the code snippet on the left and you don't have to understand all the syntax, this is just maybe focus on the high level ideas, Feast makes it really easy to declaratively write feature definitions. And these feature definitions will then be turned into very simple pipelines. So this is a pretty simple example. The use case here is you have a bunch of data corresponding to drivers. You have latitudes and longitudes. And what you want to do is set up a feature, which is the geohash of the latitude and the longitude. Now, if you did this without Feast, you would have to probably write two different data pipelines and make sure that this function, geohash, is called in both of the different pipelines. But in Feast, you can just declaratively write this feature definition once, where you import geohash. You refer to the data, which is captured in this feature view. This latitude and longitude, you basically wrap that feature view in this geohash function. And this will get turned into a very simple pipeline. And specifically, what that means is when you want your features, either for training or for inference, all you need to do is query Feast. And Feast will make sure that this transformation is actually getting executed. So that's what I mean when I say this definition is kind of getting turned into a very simple pipeline with just two inputs, one output under the hood. So this kind of transformation logic allows you to focus really on the business logic of your use case to define features and not worry about all the thorny data issues that come up underneath it. The second problem that we talked about was consistent data access and the fact that there can oftentimes be inconsistencies between your training and serving pipelines. So Feast helps solve this by providing a unified serving layer to query from. And this helps solve the problem because Feast subtracts away any of the concerns about consistency. All you need to do, as we saw in the previous slide, was define features once, and then we ensure, as long as you query from our unified API, that all the transformations, all the logic in executing and creating a feature are executed both in the training use case and in the online serving use case. Moreover, because we have this unified serving layer, it also provides some additional benefits. For instance, it can help you meet your latency requirements because our serving layer is very performant. This means you don't have to have an ML engineer take a whole bunch of different data pipelines. And for each one, use a bunch of tools to optimize it heavily. Instead, you can just put this stuff all into Feast. And then Feast will make sure that your serving layer is actually performant. And you don't have to worry about all of these different performance requirements. So the last thing I want to touch on on this slide is the fact that Feast is bridging the gap between your data and your models. So another thing that Feast stores are capable of doing, which isn't today in Feast, but we hope to add as a feature pretty soon, is data quality monitoring. So a problem that you can imagine is you've released a model into production, and it's working on real-time data. But over time, this data distribution might actually be changing in such a way that's really hard for you to notice immediately. But because Feast is this bridge from the data to the models, we can just insert some data quality monitoring tool in there. And this will make it really easy to understand over time if your data is, in fact, degrading in some way or other. And the third challenge we talked about was a duplication of effort. So as I mentioned, different teams might not collaborate easily. Their silo development and Feast help solve this by presenting this common interface through which different teams can write and then publish their features. So once you publish your features, these features get written to a central repository. And all the other teams in your organization can then query that repository, take a look around, see what features have been defined, how those features are being used in models. And this leads to very easy, basically, discovery of all the other features, and easy reuse. Because you can easily just pull in one of the other features that another team has defined. Today, Feast doesn't have a UI for easy discovery, but that's definitely something we would like to build, so that you can scroll through and just take a look and be like, oh, here are the 100 other features that some other team has built. So now we've talked about how Feast and Feast stores can help solve a lot of the feature engineering problems that we talked about initially. And finally, we can end by just taking a quick look at some of the core concepts and a little bit of the basic syntax at Feast. So over here, you'll see on the right a code snippet. You can see three basic things that we've defined, an entity, a data source, and a feature view. So a data source, of course, is pretty self-explanatory. This is where the raw data you're talking about is housed. The next question, maybe, is what is an entity? So to answer that question, you have to think about, first, what a feature constitutes. And a feature is really a measurable property of something, generally. And that something is what we're calling an entity. So if you have a feature, say, like the number of chips a driver usually takes, the entity in that case is a driver, because this feature is a measure of that driver. And so we've simply defined here this driver as an entity. Then we have our data source, as I mentioned, which is, of course, coming from BigQuery. And then finally, we have a feature view. Feature view is just a set of features that are grouped together, because it makes sense. In this case, this feature view is a property of an entity, which is specified as a driver entity. You can see the input is specified as this BigQuery source. And then there's a bunch of features in this feature view. They're not defined here, I guess, because Feast can do some inference under the hood and take a look at the schema of the table, and then infer the features from the schema over there. The next thing I want to quickly showcase is building a training set. So in this case, the way you would use Feast is say you have a bunch of events in some CSV. These events typically have timestamps. Say it was like, oh, this trip from last week by this driver. If you want to attach a bunch of features to that, what you would do is simply call getHistoricalFeatures. You would pass in these events. You would specify which features you're interested in, which in this case are two different features, as you can see, defined in this array. And then this will just return to you the features that you want. And you can plug these in to a model to train. The last thing I'll touch on really quickly is the online inference path. And in this case, you can see the call simply gets switched from getHistoricalFeatures to getOnlineFeatures. Again, you specify the features you want. And in this case, you also can tell Feast exactly which entities you care about. So in this case, if you're trying to get some features that refer to a driver, you probably want to tell Feast, look, I'm interested in the driver with this ID, 1,001, or the driver with this ID, 1,002. And Feast will simply return those features to you. And you can see printed there the features that you might want. And the last thing that I want to go over quickly was this architecture diagram of Feast. This is basically suggesting that Feast is really supposed to sit between your data and your machine learning models. And the goal is really to make it very simple to take this raw data you have and really start putting it to use when it comes to training your models and serving your models in production. And within that central figure, you can see all the different components we talked about. You can see the offline, the online stores. You can see the transformation engine, which does some of the on-demand transformations we talked about. There's the serving layer, also a little bit of the data quality monitoring that we briefly touched on. And yeah, so this is where Feast might fit into your stack. And that's it for Feast. And I will turn it over to my partner to talk about flight. Can I switch? Hello, can I hear me? Yes. Hey, thank you, Felix, for great introduction on feature engineering and insight into Feast. So carrying on from where Felix left, essentially, Feast is designed to fit in between training and the data sources. Where flight is essentially an orthogonal system, usually, it's used for automation of any end-to-end workflow that you have within your stack, for example. And in machine learning, these end-to-end workflows tend to be you get some data, you engineer those features, you train a model, then you validate the model, and then you deploy the model to some sort of serving system, or sometimes you do batch inferences. And when you actually, on the other flip side of running the model is you get ground truth, then you want to make sure that the model's not really drifting. So you want to do model monitoring, and you build some other metrics that you want to constantly observe. So that's where flight really fits in. So flight is an Uber tool that allows you to connect different pieces. And when we are also part of LFAI, we contributed to LFAI from Lyft in February this year. It's been the first few months, and we started looking around for options to integrate within the LFAI ecosystem. And that's where flight and feast came together. But before we jump into the integration, let's try and understand what are the building blocks of flight. So in flight, the basic building block is something called as a task. So a task is essentially a fully declarative, smallest unit of work within flight. It's versioned, it's language agnostic. It's something that executes. In the back end, it actually maps to a bunch of different plugins. So one plugin, for example, could be Executor Container. Another plugin could be Executor Part. Another plugin could be Executor Query on a data source-like BigQuery or Redshift. Another could be just Execute on a hosted training system like SageMaker. It's one of the interesting characteristics of a task is it's extremely strongly typed. That means flight understands that this task, even though it's a black box, I just explained completely different disparate plugins. But from flight's point of view, it's an execution unit with some set of inputs that it understands and it produces some outputs. And that's what we call as a typed interface. Once you get a bunch of these tasks together, you can just create a data flow between the tasks. And this data flow together can be wrapped into an entity that we call as a workflow. A workflow essentially models the flow of data through various tasks. And it itself is composable, so it can get composed with multiple different workflows. It itself has a strong interface, because you can parametrize the workflow. You can pass data into it and read it out of it. It's a completely declarative system, because it's just putting together units of execution. So it doesn't exist in reality, but it actually exists only in declaration. And what I mean by that is we'll quickly look at the programming model for flight. And sometimes people think, oh, workflow is a function. But it's not really a function. It's a DSL that converts from Python into actual protobuf representation, which is stored in the back end. We do have DSLs in Python, Java, and Scala. And anybody can come and write new DSLs. The Python DSL is further imperative. It supports imperative typing and imperative programming. So you can actually create new DSLs on top that fit your companies or your product's needs. The typical user journey in flight is essentially you start with an idea. You write the code. You test it locally. And usually you write one task at a time. And you test it locally. If you think your task is working well locally, but mostly most of us don't have enough of a compute power. And nowadays, we have access to the cloud. So you want to take your task and scale it to remote. So you can just take that one task, push it to remote, execute it at scale. And if you're happy with that, then maybe oftentimes we start with like, we want to train a model. We start with the feature engineering work. And once we have feature engineering done, we select a model and we try to train that model. So it becomes a workflow. So that's how you compose a workflow. You train that locally or you push it to remote. Once you're happy with that, you can productionize it. Run it on a cadence, run it based on some events, triggers, or just ad hoc. One of the other things that it does is in production, it's very critical that you can operationalize your automation. That means you want to know when something failed, when something produced an output, and also go and look for all of those outputs. And the last bit that happens usually is that once you have something running in production for a while, you want to go and look at the previous results. Maybe you're debugging some use case. Like, oh, yesterday I got this result from my execution, from my model that's serving in production. How did that model get created? What was the data that was input? Why did it generate those results? So it's very easy to do that with flight, because flight tracks, it's like having a debugger sitting constantly on your process. It's tracking everything that's happening. Another thing because of this extensive tracking is that everything is replayable and everything is reproducible. It comes with a guarantee that you can reproduce almost anything that runs in flight, as long as you, of course, you follow the constructs. If you create side effects, then those will break the reproducibility for the side effects. So as I told you, how do you write the code? So our task is essentially, if you look on the right hand side, the task is Python code, because I'm just showing the Python SDK over here. It's essentially a Python function. And if you look at the typing on those function typings, that's necessary for flight, for flight to understand what types of data does it accept and what does it produce. But besides that, there's a small decorator on the top that says that task. That's it. Flight will take that one function, convert it to a representation in the back, and execute it. On the function, you also write whether you want caching. Maybe caching is what we call as memorization, when you run a version of the task with some set of inputs. And if you rerun it with the same set of inputs we can, instead of running it, we actually just produce the output. And this does not just work locally. It also works in distributed setting. And in the distributed setting, it actually works across different teams within an organization, if they share a task. And we'll talk about sharing in a little bit. And so of course, you can execute all these tasks locally. And if you look at the second task, actually, that's really interesting, because the task actually is taking a pi-spark data frame, so it's actually a spark function. And we can talk about it a little bit more in this slide, because you have a task. And I said you want to run this now remote. You were running, let's say, locally with one CPU. You may want to use two CPUs, or you want to use more memory, or maybe use GPUs. You just decorate the task with those attributes. Or in the spark case, you decorate this task with the spark attributes, saying that, hey, run it with spark driver memory of x, y, z, four executors, and so on. And flight would dynamically create an FML spark cluster, run the job on that, and kill the cluster at the end of it. Now, that has a huge advantage, because if you think about it, every user who's running the same job may have different versions of the libraries you don't really need to manage those clusters. Those clusters are managed by flight independently. And you can experiment how you try out different libraries or different versions of a model. Like maybe one, you're using TensorFlow 2, and somebody else was using TensorFlow 1. There is no coordination needed. You just have to pip install TensorFlow within the container environment, and flight will take care of the rest. Also, actually, I never showed. At the bottom, on the right-hand side, after the two tasks are workflow, there's the decorator called workflow. That's the DSL. That's actually Luke's Python. It's exactly like a task. PIFs like a task. It runs locally. But in production, it never really runs. It just gets converted to a specification. And the bottom is it shows also that you can optionally create a schedule or get notification when something happens to the workflow. Let's say it fails. You want to get notified that, hey, send me an email if this workflow fails, or if this workflow succeeds, send an email to some XYZ. Once you have this workflow and this task, you want to ship and execute it on the cluster. For that, flight comes up with a bunch of commands. And for example, it's called a packaging. One of the commands is called package. It takes your existing code, packages with the container. Packages everything together and ships to the flight remote. Now, once it is in remote, you can use flight kit remote API to start executions. Or you can use the UI. Or you can use the CLI like Flex CTO to start executions. Because of a very expressive API, you can almost do everything generatively in these interfaces. And we'll look at that in the demo. And here's an example of what happens in the UI. So in the UI, after you've registered, you see some versions. And you see a launch button. And if you hit the launch button, it actually automatically generates the form for you. Because it knows the interface that's required to be satisfied for an execution to be launched. And as it is type safe, the launch form is also pretty type safe. It will ask you for an integer. If you actually need a JSON input, let's say, in Python, which could map to a data class, it automatically understands the schema of the data class. And it will create a launch form matching the schema. And the UI, of course, because it is a statically declared workflow, it can understand graphs of execution dependencies. And it renders the graphs in the UI. In this case, this is a graph from a real world example that we have. I've blurred it out so that you can see the names. But in there, you see some boxes embedded in the main top level graph. Those boxes are called sub-workflows. So a workflow itself is composable of other workflows. And a workflow can also be generated dynamically. So a workflow can be generated programmatically at static time, get compiled, ship to the server. But in the server itself, if you can actually, at runtime, use the inputs and generate a new workflow and recompile and rerun it. And this allows dynamic expressiveness within the workflows. The UI also captures error traces directly from the code and shows them in the UI. This is very useful in debugging your automations. A few other concepts that are very useful to understand with Flight are projects, which is essentially nothing but a logical grouping of multiple workflows and tasks and launch plans and so on. It was developed initially. So Flight started at Lyft about five years ago. And it was developed for the entire organization as a Hosted Central Service. So it has its roots at being multi-tenant for an entire organization. And so project is the multi-tenancy primitive within the system. Moreover, when we started, we were trying to train models and also ship new models to production. So we quickly realized that having one schedule on an automation is not good enough. You want to have multiple schedules potentially sometimes with slightly different parameters, sometimes because we are training a model with slightly different parameters and running a shadow experiment or running an A-B test and so on. So Flight allows that. And that concept is called as the launch plan. We also realized that when we were debugging workflows, we wanted to just execute one task at some times. With those inputs, maybe vary the inputs a little bit or change the size of the cluster or the memory footprint and so on. And so Flight allows you to execute one task at a time. And the functional model allows you to make this possible. Another problem was many of the processing systems that use Flight run for hours or days. And finding a bug at the end of a process is really irritating. So that's where we came up with the idea of static DAC compilation. So all DAGs in Flight are statically compiled, irrespective of the language. They're compiled in the backend server. And it asserts that when you put it together, the data flows correctly. Now, it can still happen that you have runtime bugs because your bug's in the code. It does not compile the code, but it only compiles the interfaces and makes sure that things work correctly when put together. And of course, it's language and framework independent. You can use TensorFlow 1.0 or 2.0. Flight does not make that call for you. Same way, you can use Spark. If you have a backend plugin and Spotify, for example, which is a top-level contributor to Flight, has authored a Flink plugin. And some others are authoring some other interesting plugins. So you can execute arbitrary executions within Flight. It's fully, you can execute everything locally. It's programmable and inspectable, and we'll see that. And everything's driven by an API. Even the UI is driven exactly by an API, so there's nothing hidden. It was built at Lyft, as I said, by the platform team. And so it was built by the platform team, and now we want to say that it's built for other platform teams. And the reason is because a platform team operates, like we are very aware of the problems and the pains of platform teams that users usually always want to extend things. They don't want to care about machines. They don't want to care about problems that happen in the platform level. And from the platform team's point of view, they want to maintain it like a service for the users so that they can deploy frequently. They can update changes easily. And so Flight has been designed to solve those problems. It's serverless. It's built for platform builders who can extend things. It has an extremely low running footprint because it's written in Go. It's incremental and recoverable, so you can actually incrementally build things. And you can recover any execution, especially in those cases where a Kubernetes cluster goes down. Now let me just recover everything, or hundreds of executions that were in progress. And Flight will recover them for you. It also comes pre-built with OR2 and SSO support, so it integrates seamlessly in your security practices within the organization. And it's fully auditable and built using GRPC. This is the... I don't have an architecture diagram. I definitely refer you guys to our Flight.org website. But I wanted to provide a competent overview because this is where it's easier to understand Flight. So Flight sits in between, where on the top is a programmatic interface or client-side plugins. And at the lower end is backend plugins. So you can extend Flight in both directions. You can extend Flight on the user layer, as well as you can extend Flight in the downstream in the backend layer. You can add new capabilities to Flight, and those are deployed independently. You can also have visibility into those backend plugins, so you can have Spark History Server, TensorBoard, all of that getting plugged into Flight. The execution by default is Kubernetes, but it also, of course, supports multiple external services. So yeah, so that gives you a quick overview of Flight, and as Felix also talked about, now let's talk about why bring them together. Because as there are, what we realize is that Flight brings in great capabilities in doing feature transforms and memoization, also allows you to use arbitrary tools to perform your data processing, and also does machine learning using training and serves as a model repository. But it does not have a registry for storing features or discovering features. It does not allow for retrieving those features online, and this is actually reducing skew between the online and offline features is extremely important as we saw today. So this is where I think Flight and Feast come very well together, and they can complement both the communities. So what we'll see is an integration overview of how we've done it today. Now we've taken a very simple example from UCI, I think it's to train a model for determining if a lesion on a harsh is surgical or not. And what we are trying to show is essentially you can do the feature transforms within Flight, and use Feast as the registry for the features, and then use Feast to retrieve those features in the offline as well as the online setting. So let's jump to a quick demo, and for that I'll use this Jupyter Notebook. It's all, the entire demo has been shipped to the example site on Flight, and I'll give you a link to that at the end of it. But as I said, the aim of the demo is to actually train a very simple Gaussian night-based model. Again, we are not trying to learn different types of model training or modeling exercises. What we are trying to do is show a demo. So we'll try to see how really we can simplify this process. And the data set over here is from UCI, let me see if we can click on it. So this is the data set. It consists of a bunch of different types of features that are available or attributes rather. And it's a well-known database. So once you, sorry, the code for the pipeline as I said is written in FlightKit, and it is actually being pushed into the Flight hosted examples repository, which consists of an auto-generating documentation as well. So it gets generated into a documentation. It talks about the data set that we are going to show and the actual workflow. So the workflow can be seen here. It consists of as we talked about a bunch of tasks and how the data is flowing through. The most important thing to look at over here is the workflow itself. The workflow talks about building, first we build a Feast feature store. Then we call a process called the featureize, which creates the features, loads them into the feature store. Further, we actually call the training algorithm to train the model, and then we try to do a prediction online. And the important thing within this demo to look for is featureize. It's doing a simple SQL query. We've taken the Horst-Collect data set and it's a SQLite DB, so we're just doing a SQL query on SQLite, getting a data frame. We are doing some sort of, we're trying to impute any of the existing, any missing features, and then adding some, sometimes missing timestamps and so on, and then storing it to Feast. And this is a demo and an example of how you can do feature engineering. Once you have those features, these features are passed in, think of it as a flight schema, but think about it as a pandas data frame. It's a higher level abstraction of a data frame and because data frames, there are multiple different types of data frames available. You can have a Spark data frame or pandas data frame and different layers of scaling available with them. For example, modin and so on. So we abstract all the data frames just by the type for the data frame, which is essentially a column not stored, each column having a type and that's called a flight schema. So we take the data frame and we actually pass it through, in this case we had future selection and then later on training a model. And model is a, we can actually look at the training model itself. It's pretty straightforward. It's a, I think in a scale on model. Maybe somewhere it was, sorry. So let's go back here and check out the code. So let's go and go to the next bit. So what we, what I've done previously is I've actually gone ahead and taken the code and registered it with flight. And so I'm going to use flight kit remote, which is an interaction system for interacting with the flight server. And you first initialize the flight kit remote and then you say, all right, I want to fetch my workflow from remote and it will tell you the version that it's executing. And we'll see, we'll look at the UI in a bit to understand more of this. So once it fetches the version, it's currently just, you can also definitely specify a version for the workflow directly within the call. But if you don't specify when it fetches the latest by time. And so it's fetched the latest version by time. Then we go ahead and take this launch and we execute it. And this execution results in an execution. As we're seeing, this is the graph of the workflow that we just started. We just talked about. We just saw, if you see, it's creating a bucket that's needed for Feast. It's actually building the Feast feature store and then calling this training system. And if you see the blue over here, that means the training is worth in progress right now. After the training is complete, I will load the historical features and call, sorry, this was the feature rise and this is the training system. The training will be in progress in a bit. And in parallel, once you have the train and once the model's ready, we'll run a sample predict and we'll end with that. So if you see, I think it's already done by the way. And it's done so quickly with so many steps because it actually has a memoization built in. So whenever we saw that there was no more work to be done, we actually just started memoizing things and we started retrieving results from a previous run. And that can be seen because of this icon, which says that the outputs were read from a previous result. You can of course go to the source execution and it shows you where the output was written. And in this case, it's pretty interesting that actually one of the tasks failed. So even then partial memoization was done and as I updated the code and I fixed it, I could actually retrieve the results. These are all the various versions of the pipeline as I was trying out and I was failing. And I was doing executions and this is the most recent execution and most recent have all been successful because I was doing a demo, but the old ones were failing, right? And you can actually go back down in time and when you click on one of them, it shows you the error at the top. It's actually, I was not getting a right feature from Feast at that point because I was passing my data type into Feast. It expected a string and I was passing an integer. So that's what it shows, but you can actually collapse this and you scroll down further. You can see that what could have happened where you'll see the inputs. The inputs is the feature store. It's all tracked in here while the serialized model because it's the predict function. The model itself is serialized using JobLib and it tells you that I'm serializing using JobLib and the outputs didn't exist in this case, but in the previous case, there were outputs and showing you all of that. So let's go back to our notebook. So now that the execution is completed, we can ask to sync the execution. It shows that the phase is complete. So let's sync it and once you sync it, you can actually access all the outputs of the workflow. So if you go back here, you will be able to see not for the failed one, of course, but for the successful one. This is the most recent one. We can always see the inputs and outputs of the entire workflow. So if you see the output was the serialized model, the feature store, and the sample prediction that we ran, which resulted in one being the predicted value. And the input was the feature store and the imputation method. So if you go here, we can actually retrieve these values again. And interesting thing, we can retrieve them in a very types of way. So I can actually construct a feast feature store completely locally. So let's try and see what that feature store looks like. So it says that the database is a SQL IDB and the bucket where it's getting stored is feast and the online DB. The model itself it shows is it's actually available as a joblib.dat file, will be available as a joblib.dat file if I materialize it. So we will go ahead and materialize it in a bit. But before that, let's actually load features from this feature from feast feature store. And I want to load, I just found one arbitrary hospital number within the dataset. And I said, let's check out the point. And so that should run in a bit. And so what's happening in the background is actually downloading the feast feature registry and feast feature registry itself is using an online feature store called of type SQLite. So it's actually downloading the SQLite file, loading all of that and all that is available in your Jupyter Notebook. So now you have the dataset available. You can load it and you saw the inference point. Now let's actually run a prediction. And if you see the, I'm actually passing in the serialized model and the features. And this predict method is actually the same predict method I used in my workflow. So I can run that locally as well. And it comes up with a quick prediction. So what happened actually, it was too quick, but it actually downloaded the serialized model, unpickled it or unjob lib date and loaded in memory. And that's actually the demo, but what happened over here, we can of course observe, as I showed you the intermediate within the workflow and you can look at all the data and you can use the UI to launch this workflow. I wanted to show the Jupyter Notebook example, but we think there are a lot of further improvements that we can do in the system and we can use the same predict method for really fast inference as well. Let's go back to the demo and we have a few more minutes. So let's, this integration was possible. Thank you from the help with the feast community and the flight community. And I think it was amazing to work, but we've realized through the integration, we realized there are some more work to be done in the integration and a few of them, like this one, I'm gonna call out a couple limitations and strengths. So when you register, the feast registry is a monolith or a singleton and flight workflows are highly concurrent. So you can run like thousands of workflows potentially at the same time. And you may be generating a lot of features or registering new features or creating new offloaded features at the same time. So if you do that today, because the feast registry is a singleton, you will overwrite and probably corrupt it, but the feast team is working on actually making multiple concurrent writes possible. But once you have the feature registry, you can read it in parallel across multiple different workflows and that actually makes it really, really simple. And since it's in S3, it's completely isolated and simple file-based. The scale is almost no problem. And the third thing is the same thing about like creating the registration, like what happens is when you materialize features in Feast for online serving, it also mutates the registry and that's also not possible to do it concurrently. And hence it's important that you manually kind of serialize it at the moment, but the feast team is aware of this and they are working on it. One possible solution within a flight could be to actually use a different registry because registry is just a file, so you could essentially use a different registry for every single feature training workflow and as the workflow runs, it creates a mutation and a new registry R-dash is created and you can use the R-dash in downstream workflows and essentially you end up with a lot of different registries. So the problem over here is that there is no unified view of the registry which is a big problem that as you grow larger as Felix pointed out, you have to have a unified view, but a benefit for this is that it's essentially infinitely concurrent and you can really scale to as much as you want. And as you saw in the flight API, you can always retrieve the registry corresponding to a specific execution or a specific workflow. So that's, I think on our side, hopefully it was useful and please ask us any questions and join both the communities. I think flight.org and Feast.dev are their websites and we are very active on Slack in both the communities. We would love inputs and it would be great to hear from you. So besides, and of course I want to thank a bunch of other people who actually couldn't present here, who are the people behind the integration and I am just like the mouthpiece. So any questions, comments, concerns or suggestions? And if you are interested in learning about flight more, we already had two talks. We have one other talk tomorrow about Harvard and how to do a distributed deep learning on flight. Please attend it and let us know. All right, thank you. Thanks.