 Hello, thank you for joining us. It's my name's Kathan and we have Katrina here with me. We both work for a company called Union.ai. Both used to work for Lyft before this, and are upstream contributors to the flight project. And I'm very excited to actually talk about our work that we've been doing with the Horovod community in getting distributed deep learning training available natively within flight. And both Katrina and I will just go over through the work that we've done, and demo and show essentially how easy and accessible this makes deep learning training on flight. So before we get started, the quick agenda is, we'll do a little bit, we'll go through understanding Horovod and deep learning, and distributed deep learning training where it fits in and how does it fit with Spark, because that's the title of the talk. And then we'll talk about flight, maybe just give you a quick overview. If you've not heard about flight, this might be a good primer, but this doesn't cover everything in flight, it just gives you a little bit of an overview and a glimpse. After that, I'll hand it over to Katrina and she'll do the majority of the talking and the interesting stuff, which is essentially talk about flight and Spark and Horovod, how it all works together to a demo of that. And then for those folks who don't really use Spark, potentially in their organizations, she'll show what's an alternative to using Spark for Horovod. And then we'll conclude. So let's get started. Let's do a quick recap. And this is an interesting graph that I found online. This was in a research paper recently or some time ago. And it's been used a bunch of times in various different talks. It's very interesting because in traditionally, if you use any other machine learning technique, some amount of data is, if you have too little data, it's not as good, but as the data increases, the accuracy increases, and then it starts flatlining if the data continues to increase. And that's because various things come into play. The models are not able to really understand the complete dimensionality of the features and many different things. But with deep learning, this changes quite a bit. As the data continues to increase, the accuracy of the model continues to increase. And this has been studied across various different models. And so conversely, the more data you have, the more your model can improve, but then more problems you have because you have a lot more data, so that means your training will take longer. If you're training one model, you're using one machine, it will take that much more longer to train your model on that entire dataset. So another solution to that is, what if we distribute the data? If you distribute the data, there are some problems that happen, but it may allow you to speed up the entire process of training the model. There's another thought of distributed training for deep learning, and that is usually not distributing the data. This is the case in which the model itself doesn't fit one machine. And that's not the scope of the stock. We are just talking about the cases in which the model fits on one machine, but the data is too vast and you want to speed up the training process. So let me do a quick and simple overview of distributed data learning, distributed data parallel deep learning. And again, this is just a simplified overview. There's a lot more details to this, and I'm not going to go into all the details, but let's assume you have a model and you have a large dataset. One way to train it is just to take a dataset and train the model on one GPU instance somewhere and at the end come up with a trained model. But as I said, that will take long. So how do you parallelize it? A simple way is to take the dataset, divide it into n chunks, let's call them shards or partitions or whatever you may, and then train the same model on those each individual partitions in parallel. And at the end of it, what happens in deep learning is you're essentially learning weights or some numbers within your neural networks. So at the end of it, you could take, and they are called gradients, at the end of one training process, you take all the gradients across all these machines, average them together, or some kind of a aggregation, and restart the training process from that average value and do this till the averages converge or reduce the changes in the average reduce. And this is called a distributed data parallel training. Basically what you've done is if parallelize the data training process by splitting the datasets into multiple paths. So how would you do this? The critical part of it is how do you do the averages? And there are multiple different approaches. And interestingly, there are two approaches that have been implemented in the systems that exist today. One of them uses a very common architecture where you have a central arbiter or observer, and what that does is the central observer at the end of every round receives the weights, it does the averaging, and then sends the results back to all of the workers, and then the workers continue on. And then again, in the next round, send all the results back to the central server and then return and get the updated results. And this works pretty well. It's really simple to understand. It's also used in practice. There's a problem with this, and the problem is as the total number of nodes increase, or workers increase, the central process becomes a bottleneck. It has to handle a lot more communication. It also is a durability bottleneck where if the central process goes down, you potentially end up losing your progress. So now you need to invest in high availability and so on. But the pro of this is actually much easier to implement because you just have to know one address. Who is my central observer? I just go and talk to it if I'm a worker. An alternate to this solution is something, and I'm just gonna kind of try and come up with the intuition behind the system, and then we'll look at, actually, we'll talk about how it's implemented. An intuition is what if I know, if I'm a worker, what if I know my neighbor, and I tell my neighbor, hey, I got these results here, it's my results, go take it. And that neighbor takes it and averages it with his own results or their own results and passes it to their neighbor. And this continues on, and if we kind of arrange this in a ring, then the person who started the process receives the updated results from the last one. And this is actually a popular technique called as ring all reduce. And interestingly, this reduces the problems of the previous one where you had one central point of failure. So this is where Horowitz comes in. Horowitz essentially makes it possible to implement ring all reduce very easily. It uses an open source library called as MPI which comes from the high performance computing domain. High performance computing, if you guys know, there we used to have the supercomputer challenge. That's what's called the high performance computing, like there's an entire computer science domain that works on solving really, really complex problems on really, really large machines. And that's where MPI originated. So MPI essentially is a case in which we have multiple, it's called the message passing interface, that means you have multiple nodes, all of them talk to each other given that they know each other's addresses. So they have to know what's my peer group. If I know my peer group, you can write very, very complex algorithms that talk to each other and it's simplified by the MPI process. Horowitz took the MPI libraries and built a layer on top of it to further simplify distributed, deep learning gradient averaging systems. And they do it by providing different types of optimizers. And the cool part of Horowitz abstracts the entire networking substrate, including like, there are many different types of networking substrates possible now, so you can use something called as NICL for inter-GPU communication. Then within machines, you can use something called as InfiniBand or you could use Ethernet and Horowitz abstracts all of that away. So then where does Spark fit in and why do we wanna think about Spark? And the question for me is like, why not? So Spark's essentially solved the problem of starting a bunch of executors, all the executors and knowing all the executor addresses in the central driver process. And Horowitz essentially piggybacks on that. Horowitz asks the driver process, hey, who are your executors? Can I get all the IP addresses? And then it starts an MPI domain within that IP address space. The problem here is though, you have to remember, Horowitz is running still using MPI. It's not using Spark native, any kind of shuffle mechanisms or something, but it's using MPI to do the already. So it is still doing a ring-based already use. The benefit of doing it this way is that Spark then allows you to do the processing and you can use MPI to do the deep learning. All right, so now we know Spark, Horowitz. Let's talk about flight, the third piece in the puzzle. What's flight? Flight is essentially a workflow automation platform that was built five years ago at Lyft and was open source last year. And it exists to essentially simplify the process of automating a workflow for either of your machine learning or data processing needs. And it's more often than not, that's actually the problem that many folks are trying to solve is they're like, oh, I have this process in which I take some data, I convert it to a usable feature and now I want to train it and then I want to take the train model and run a bunch of batch predictions. So flight takes that easy. That's what we call as the workflow automation platform. But to understand flight, you have to understand a few concepts and we'll start with the first basic concept. Flight consists of a process called as a task. A task is the smallest building block. It's essentially a unit of work. It, there is no definition or prescription of what the unit of work is. It can be anything, any arbitrary process. For example, it can be a Python function or it could be a distributed Spark function or it could be a web API call or it could be a SQL query or a query that runs on one of the hosted data stores like Snowflake or Redshift or so on. What happens is it's a task gets mapped to a backend plugin and that's at the execution point where it actually maps to the right compute engine. A task is completely declarative. You declare it ahead of time, it's versioned and it has strongly typed interface and this probably is one of the most important property of a task because we understand, flight understands what data needs to be sent to the task and what output it produces. When you take multiple of these tasks and chain them together and in a pattern of producer-consumer, so essentially a task produces some output and another task consumes that output, it becomes a workflow. So a workflow is nothing but an ordered set of tasks which model the data flow through those tasks. A workflow is purely declarative. It also has the same properties as a task versioned and completely decorative and composable. You can also write it in. What we have done with Python FlightKit is essentially made it possible to declare this workflow in simple Python. It looks very Python functional like what essentially gets converted to a specification in the backend. There's also another library in Java Scala contributed by Spotify that provides the DSL for writing this in Java. The Python FlightKit also allows you to write your own DSLs if you want to use Python as the medium for compilation, but you can of course use this up the base language and write new DSLs in other languages. A typical user journey when you're using Flight is, the idea of Flight was to abstract infrastructure and make users really focus on business logic and connect pieces. So a user usually writes the code and a task at that and they can run it locally. If they want to scale that task to lots of machines, they can send it to the remote, run it on the hosted server, compose a workflow by taking multiple tasks, run that locally, run that remote. Once they're happy with their work, push it to a production environment and make it completely indestructible in some way because there are recovery mechanisms or retries or run them on a schedule, run them on a trigger and also get monitoring metrics so that you know when things are failed. Also finally, this is oftentimes what happens is once you have things in production, you want to look at the past and you want to make sure that what happened was, what were the reasons why some thing happened. For example, a prediction that was given by a model, how did that model come about? It was trained by this pipeline at this point in time with this input data and you can actually flight will allow you to introspect everything from that past execution. So let's take an example of how do you write this code. So in this slide, on this slide rather, on the right hand side, you can see two tasks and a workflow and these two tasks are very simple, they are simple Python code. Actually, the second one is not really a Python code, it's a distributed Spark code, which you can run locally and the only way I know that is because it takes in PySpark.dataframe, which is not really a simple data frame, it's a resilient distributed data frame. So a workflow though, if you can see it can actually put together two different types of tasks, one that is a simple Python function works and produces a pandas data frame, produces a pandas data frame and the second that takes a PySpark data frame. The only thing that you need to do is after writing your regular Python function is annotate or decorate your tasks with a simple decorator and that's where flight kicks in. Flight knows that oh, this is a function that the user is interested in potentially executing it on the remote cluster. You can also enable caching by simply adding a couple parameters to the decorator, like in this case cache equals two and cache version and then as shown at the bottom, you can execute it locally just like executing a regular Python function, albeit you have to pass only keyword arguments supported. All right, now that you have, you ran it locally, everything looks good, let's scale it. So how do you scale it? When you're running it remote, maybe you want a GPU instead of running CPUs. So you can specify those resources in the same decorator so you can update the decorator to now have resources as well. In the case of Spark, right? I said that the second function is Spark. You actually tell Flight about it that I want a Spark cluster with this kind of configuration and we try it on it in case of failures and Flight will make it run. At the bottom, you also tell Flight how do you want to schedule this workflow? So sometimes you wanna have multiple schedules on the same workflow or you may want to run it every few days or get notifications on failures and that's what it shows at the bottom. And of course, I just split this across two slides, you would do this in one go and you can run everything locally. All right, so now you have the code, you actually decorator it, you're ready to run it remote, how do you do that? You have to go through a process called as registration and the process is essentially packaging your code from local and uploading it into the server. So FlightKit, which is a Python SDK, comes with a handy CLI that does the packaging and then uploading to the server. Now you can execute it in multiple ways, you can execute it using a CLI, a UI, or actually use an interactive notebook environment to start an execution and also introspect the outputs. Important thing to notice when you execute it, Flight is extremely types safe, it knows about all your types as it knows about the inputs and outputs, so it can easily generate a UI form for you and which we'll see in the demo, as well as verify and validate the inputs that you're passing into the system. All right, here's an example of launching an execution. So in this case, in the left-hand side, if you see in the left-hand side in the window, if you see there are versions, those are the various registered versions. Flight is immutable once you register a version, that version exists forever till you actually have some garbage collection policy, but you cannot really delete a version, you can just register a new version and so everything is recorded by Flight and it continues to exist. And at the bottom it shows executions, it shows you errors in line and at the top on the right-hand side, you see the red marked box that actually shows you that you can launch this workflow and once you hit the launch button, the launch form on the right-hand side pops up and you can fill it in and hit the launch. And we'll see this as a demo. Also, statically or when things are running, you can see the entire graph of execution. This is the power of having a statically compiled workflow, it essentially converts to a execution DAG. One of the other advantages of using Flight and FlightKit in concert is that essentially there is a way for Flight to grab the most important part of an error output and show it directly in the UI. So in this case, it shows the stack trace directly in the UI and in here, for example, the message at the bottom actually tells us that it was a user error and we didn't have the right set of resources or something set up. All right, so a few more things on Flight. Before we start using it, we need to understand some concepts like projects, Flight started at Lyft and it was operated as one central service for the entire company and across all other companies that use Flight, they operate, there's one team that operates it as a service for the entire company. So it was built for multi-tenancy within a company. It allows, it comes up with a project of, oh, sorry, a concept of projects which is a logical grouping of tasks and workflows so teams can operate independently and provide the self-serve type of an environment for it. It has a concept of launch plans. This is again, when we were working, we figured out that we found that we needed to launch the same workflow multiple times or run multiple instances of the same workflow on different schedules, potentially with different inputs. And for example, in training, you may want to run another instance of a workflow with slightly different parameters just to run an AB test. We also found that when you're running a workflow, oftentimes one task fails and you want to debug that again and again, so Flight has a concept of memoization and caching, but it also allows you to run that one task independently and this helps you to debug and build iteratively. We support local execution through our rich SDKs and everything is API-driven. You can use the GRPC APIs to run everything and write your own UIs and interact with it through your own services. Another thing is that Flight was built, as I said, at Lyft by platform team for the company. So it is built by the platform team and was open sourced and we love other platform teams and we love working with other platform teams. It's great for platform teams to contribute to Flight and use Flight essentially because it allows them to offer a serverless experience to their users so that you can centralize the usage of all machines in one service and that way you can trade off a cost with performance in various cases. It's also built from a point of view of extensibility, so platform builders usually love it and the reason is because they can add their own ways of doing SSO, their own ways of doing specific plugins that they want to connect to their own services and so on. It's built to be incremental. We understand that there are infrastructure failures that happen every now and then and Flight will recover from those failures. So this keeps your users happy and hence platform folks love us. It is also built in a performance goal type safe, easily deployable, uses protobufs, so always maintains backwards compatibility and also helps has a low footprint. It's secure by design. It comes with SSO and OR2 integrations built in and it's completely built in GRPC. So all the APIs are documented and extendable. Usually we see an architecture diagram of Flight. I decided to do something different. We are showing a component layer cake diagram. The purple is all the components that are already built into Flight and are available. The way they are layered is essentially how they layer on top of each other. So Flight console, Flight CTL, Python SDK, Java SDK are user-facing components while Flight admin is like the control pane while Flight propeller and so on are the backend components that usually users don't directly interact with. But and this also is how things can be deployed. The control plane and the backend data planes can be deployed independently. Also, Flight interestingly can be extended in either directions. You can extend it up in the upper levels. You can add further simplifications on top of the SDKs, make it easier for your use cases as well as add simplifications at the bottom layer, extend it and improve it at the bottom. So that's a quick introduction about Flight and Horobot. But let's jump into the meat of the talk and I'll have Katrina talk about it. Thank you. Yeah, okay. Thanks, Kati, for that overview and introduction to Flight, Spark and Horobot. So let's see how all these different components come together and how we would use them. So using them is pretty simple. All you have to do is install this FlightKit SDK extension here, which essentially builds on the existing SDK that Keith had mentioned, but brings in some different Spark dependencies and allows you to write tasks in Spark. Let's go over what the interaction of all these different components would look like. Sorry, okay, is that better? So you can see here our happy face representing our user. They decided to go ahead and create a Flight workflow execution. And what happens after this is Flight resides on a Kubernetes cluster and Flight will go through and iterate through all the different tasks to find a new user's workflow. Once we come to a Spark task, what happens is Flight uses the native Spark operator on Kubernetes in order to go ahead and create a Spark job, which brings up an fmrole cluster that includes a Spark driver and executors that have been specified by user's code or with parameters specified by user's code. The environment in this case is locked in a container, which is defined at registration time so that you can have idempotent executions, that you can launch multiple times for the same Spark task definition. And this model also gives us isolation per execution, which enables resource isolation and also allows for users to import different libraries and dependencies without having to worry about different conflicts with the existing clusters since we bring up a container for every single execution. Let's take a look at how we would go ahead and write a Spark task. So you can see here the syntax is pretty familiar and similar to what Kathan showed earlier. We use our task decorator as before, but this time we add in a Spark config where we configure different parameters, such as the driver and the executor resources. This all results in a container which gets built with these locked in parameter or with these locked in config options that gets submitted upon registration and then is used at execution time. In the future, we can actually work to run Spark on external clusters as well, if that's the setup that your organization depends on. Flight has the native web API plug-in, as Kathan mentioned, and it's easy to submit requests to external clusters if that's a requirement. Let's take a look at how all this kind of goes in together with a fully fledged demo. So in this case, we'll be using a demo from the Horovod team that they produced, which trains a model to predict sales data for the Rossman drugstore chain. This workflow just modifies the existing Horovod code by adding in a flight task, which goes ahead and downloads the data from the Cable Competition dataset where this originates from, uncompresses that, and then uploads it to a flight directory, which is a native flight type, which you can use in either running flight locally on your machine, or if you have a hosted deployment of flight, this integrates with cloud blob stores to upload your files so that they're remotely accessible. The second task consists of the bulk workhorse of this model training workflow. You can see here, we define a few hyperparameters that we use for building our model, and then we have yet another task. This one has that familiar Spark config that we talked about earlier, and then what this estimate task does is actually consists of three kind of separate tasks, which if we wanted to, we could further decompose into separate flight tasks for improved isolation and memoization. The first stage of this task includes data preparation, which I won't get into, because that's not really the purpose of this talk, but what this does is it handles that input training dataset and prepares it and cleans it up some so that we can use it to build an estimation model. The second stage of this task consists of the actual model training, as we see here, and the way this works is we use the Horovod Keras estimator, which essentially accepts any Keras or TensorFlow model and runs that on Spark and handles kind of the inner operation between gluing Spark data frames to the distributed data parallel training and building a model that can accept data frames and predict a sales data on them in the future. And then finally, the last stage of this task consists of the final prediction, which produces a prediction file on the Voldat test data, which we can use to submit to the Kaggle competition if we desire. We can see here, here's our workflow, again, with a familiar decorator, and you can see there's two tasks, that data download tasks, and then we pass that data directory into our, oops, into our second task, which runs the Eskimate. And you see all this code for, and all the examples here are checked in and available on our official read the docs, and as you saw just now, linked to all the flight code as well too for documentation and probability. So let's go and give this workflow a whirl. So this is the flight console that Kate mentioned earlier. You can see we have different versions of the workflow that we've registered, historic execution data, you can see where I was trying to get the demo working and then eventually succeeded. So let's go ahead and launch a workflow execution. So we'll just relaunch an existing workflow execution. Take a look here. So here are our hyper parameters that we've defined before, as well as a working director if you wanna override that. I'm not going to because we're using caching for this example and I wanna make sure that it runs within the time for this presentation, so we're not gonna modify anything. So let's launch that. And as the flight platform works on executing this workflow, the results will be piped into the console and you can see them as they come in. And boom, there, our workflow is done. In this case, it ran super fast because we were using that memoization that Kate then mentioned. So all these results have been pre-computed so we didn't actually have to bring up any spark job or Kubernetes containers. But we can take a look and see what the spark flow looks like now that it's succeeded. So we have these two nodes that represent those two different tasks in our workflow. The first is that data preparation task and then we can see the outputs of that fed into our estimate task in this kind of visual graph workflow overview. We can go ahead and take an individual look at our individual tasks. You can see here that succeeded. We can take a look at the inputs that we passed in here including this hyper parameters as well as that work directory that included where we downloaded the data set, our outputs where we uploaded our prediction and a test definition. And what we can actually do is look at the source execution in this case, the one that wrote to the cache in the first place. And if we wanted to, we could take a look at the logs there and let's say, if you have a Spark history server integrated with your site deployment which is very straightforward to do but we haven't done in this case, you can actually take a look at the Spark history server logs there as well. So this was all great and happy but let's see how flight kind of helps you debug and let's say a scenario where you're iterating as a developer and you're trying to understand where things might be going wrong. So let's take a look at a previous failure. So you can see here my first task ran successfully it read from the cache, the second task failed. So we actually have inline stack traces that make it really easy to see what went wrong. So in this case, you can see that we didn't have enough resources to run our job which kind of represents a system level failure but this doesn't necessarily mean that our code is incorrect. So what we can do is if we want we can actually go ahead and recover this workflow which allows us to essentially resume execution from the last known successful point. And even though we have caching enabled in this case you don't necessarily need to have caching in order to be able to recover a workflow execution. So this really kind of shines of helping you improve your cost efficiency in the cases where you do have these kind of system failure scenarios. Great, so that's great and all but sometimes Spark isn't necessarily the right tool for your organization and this can arise in several different circumstances. Sometimes you might have data scientists with different backgrounds who aren't necessarily familiar with Spark and it has a kind of steep learning curve. So using Spark it can be difficult. But sometimes Spark can also be expensive in the case where we saw we had our data preparation phase and then our model training phase. A model training phase is GPU intensive and requires that you have nodes allocated to GPUs but that data training phase doesn't actually need those GPUs. So they're sitting idle for the duration of that workflow. A lack of visibility. Horibod and Spark run in different processes so they have different logs who can be difficult to kind of debug when things go wrong. And sometimes Spark is just like not exactly the tool you need for your job. If your data is unstructured, it can be easy enough to just use different functional methods like map in order to process and build your data training model. So in that case, we'll wanna use the MPI operator which Kate the one over earlier. As you can see here again installing MPI operator for use in flight is as simple as installing that Python library and making sure that you have MPI enabled in your flight backend. And there's a backend plugin system that makes that really easy to manage within flight. And the MPI operator in this case actually, oh sorry I should be presenting. MPI operator in that case handles essentially communication between all the nodes and setting that up so that you can run a distributed data parallel training seamlessly. Let's look at an example scenario where you might use the MPI operator. So let's say your data exists and data warehouse somewhere, you might wanna use Snowflake to kind of go ahead and process that data, query it, do some preparation, and then you would use Horovod to run distributed training on that. So flight really shines here because these different individual units of work correspond to different kind of extensions such as say Snowflake or Horovod and flight handles kind of the discretization of all these different tasks. And then in this case you can see where we run the Horovod distributed training. In that case maybe we need GPU allocated machines so we can run those there, but for the other tasks we don't since these machines are extensive we can save money by running them on non-GP enabled machines. So let's see what this looks like in practice. So we'll take a look here at the code that will be checked in soon. So we have for this example what we're doing is we're using the famous MNIST data set to train a machine learning model to recognize a numeric characters. You can see here we have a single task this time annotated with the MPI job or with the MTA with the flight task decorator and we have MPI job specific config to handle initialize or handle work starting the MPI job. So we have our training tasks, we initialize Horovod, we download the MNIST data set and then we initialize a Keras model using the Atom Optimizer that will build and train. And then we use Horovod to run training in batches across our training data. And you can see here we have this checked to Horovod rank which ensures that even though we have all these different like machines running our distributed training only one should actually write the results and flight gives you tools such as the Ignore Outlets exception which allows you to make sure that you can safely not return any results without that affecting your workflow success. After we're done with that we'll go ahead and save a checkpoint and serialize version of our model and return that and this will be all uploaded for us. So let's go ahead and trigger a workflow execution for this. In this case we'll actually be using a different interaction model of flight and we'll use the programmatic API that Keith mentioned earlier. So is this big enough? So here I have a Jupyter notebook set up and you can see here we download the MNIST data set again in this case we'll be downloading or we'll be using the test images to see how well our model performs. We can cruise through the test images here and see if these all look like the numeric characters that we're familiar with. So let's go ahead and trigger a flight execution. So this is pretty simple. All you have to do is initialize a flight remote client. In this case you can see that we're initializing with a project which is flight snacks we've used previously in the development domain. And we've passed in a config file which includes different authentication parameters and other settings in order to open that client connection. Then we go ahead and we fetch a launch plan which is that kind of partial with the bound inputs that we use to trigger a workflow execution. And then finally once we have that launch plan we'll go ahead and create an execution with the following inputs with batch size and buffer size and data set size specified. So let's go run this, see how that works. Cool, so we created this execution. Awesome, let's see how that looks on our console. So you can see that execution immediately ran and because we're using caching again it succeeded instantaneously. But you can see here we have that MPI task, simple workflow overview, awesome. Let's go ahead and see how well our model did. So back to our Jupyter notebook we can go ahead and sync that execution state and fetch those outputs that the execution model produced. And see if we can load a model with that. So, there we go. Great, so we downloaded the outputs to local directory. Now let's go ahead and load a model with that serialized model that we output and see how well it performs when we feed in the test data. Voila, our accuracy is at 98.68%. So good job us. And if we want we can go ahead and look through a few of the test images here. To see how our model actually predicts. You can see here is seven. We predict seven, two, two, one, one, awesome. Model looks like it's working. So in the future there's definitely ways we can improve utilization and cost efficiency on the flight platform. This can include using a streaming type transformers to essentially stream data as it's prepared into the training phase so that we're not sitting idle waiting for all of our data to be processed before we begin training. But to conclude, flight really adapts to your workflow. It's extensible, all these different plug-in models help you run Horobot on Spark or Horobot with the MPI operator but still maintain the same utility that you expect from flight. And this includes that unified user experience we saw with the console and the debugability and so the remote API. Memoization, so all caching which helps beat along all of our training tasks. Recoverability in the cases of some failures where you wanna be able to relaunch your workflows and reproducibility which is a really kind of like permanent problem right now with machine learning. So you can track your data lineage and get all accomplished through the flight. Before we end plug for Hacktoberfest if you're interested in the flight and you wanna ease into contributing we'll be participating in Hacktoberfest so you can go ahead and check us out on FlightWorksLock and visit the channel there. And that's it, thank you so much.