 Yeah, that's fine. Okay, so the topic of this talk is products for the ML 1 Q-Flow. So, we'll jump straight to some of the takeaways. Basically, these are the messages that I want to convey to you in this talk. Okay. One, opening an ML platform on the Q-Flow makes a lot of sense. And we'll walk through why that worked at Goja. And two, we're going to introduce Feast, a feature store that we call the Project of Open Source. And what we're going to show you is that through demo, using Feast with Q-Flow allows us to, allows data scientists to quickly entrain our models and deploy them into production. And this one we're going to show you is the quick highlights of how that process works. We've just been able to do two stores, and then you trade them all in the webcast and serve it with the Feast server. So, a quick agenda. I'll cover the Goja server things, the evolution of our tech stack, as we did at our platform. And some of the decisions we made are why Q-Flow has made sense, and Q-Flow has made sense in that case. Then we'll walk through a quick demo with Chris on Feast and Fearing, one of the new components of Q-Flow. And we're going to deploy a model there, with Jimmy and I go back to Q-Flow. And then Chris will cover some of the latest developments on the Q-Flow side. So, if any of you don't know, Q-Flow is an open source and all of that form, all the public Kubernetes by Google Cloud AIT. It's really quick to have a source, and you can find it online, Q-Flow.org. I'm going to assume that most of you already heard of it at Feast, so I'll dive into the specifics. For any of you that don't know Goja, I just quickly want to give an introduction again. Goja is a technology startup from Indonesia. We're most famous for ride heading on motorcycles, so it goes ride, but we've brought it to many other services and products, like the food delivery, the car, driving the cars, the campaign, digital payments, just services like the unboxing, they're saying that many other products and services. In total, we have about 18 different products and services, and all together, the 4.2 track with the goal of solving every work they need that you have. In Singapore, there's a new ride heading at the limit, and cars you have to have more services to. Just a little bit about our scale. So, the previous products we have are, at first, the ride heading. We have over 2 million riders, and the first is one of the other many bookings, every month. Any given point in time, just hundreds of thousands of riders online on our platform. For food delivery, we have one of the largest food delivery services in Southeast Asia, with over 400,000 merchants on the platform, so goat meat products, goat meat, also one of the largest in the world in Southeast Asia. So, the scale is pretty large, and the ability for us to impact these systems. The ML and data science is immense. And when it's not just these products, of course there's a long tail of making other products in the world, and there are many visual touchpoints for ML and data science. And I think a lot of you have seen this slide before. It's one of my favorites. It's a deli in Jakarta, where people represent the person they picked up or dropped off in the city, on a motorcycle, and I think this illustrates just the volume of data we have, and how data is important to our decision-making. So today, a lot of these decisions and transactions and bookings are done through machine learning and dosing systems, not considering the localization. We didn't get there. We originally had to start an open platform from somewhere, with making the decision how to go after the platform to support all of our discussions, not just the submiter's cases. So when we originally started, we didn't originally be using a data service. I think Yann was illustrating a lot of what exists today at Google. Some of that did exist in different boards earlier. It's hard to know when you start on VMs, do you go from scratch, do you use Kubernetes, something like SageMagnet, or any other Google services. But when we started, we looked at the problem space. And we did kind of what the problem space are about this, if you look at the business impact versus engineering complexes. So what I mean is that as the business impact increases, some of your core services like recommendations like food, for example, or even it's like cards on the page of the application, search and pricing, search price. These are critical systems that versus hundreds of millions of transactions around the world in some cases which cost you a lot of money. So obviously, that'll have a lot more engineering requirements. And you also have a long tail of other smaller tasks that require a lot of data science. So let's say an analyst that just wants to train them all to do some forecasting. It's not going to be going to production. But he wants a different experience. So that one should solve both of these cases. So the data science side, we want high abstraction. We want productivity. But most of you want abstraction. So user interface or a network or something that hovers that person to do more. But on the engineering side, the requirements are different. You often, especially in a company like Berger, we have hundreds of engineers who are made of different systems. You want to be able to do that. But all that supports the non-heavy case, the age cases. So you want to fix productivity and production at scale. So the platform needs to support both of these cases. And you can get the requirements looking at the problem space. We wanted to support at least one or two of our people business systems. So we focused on one of these in the striver allocation. And we knew it was going to be a complex engineering system. And so we were really leaning towards something that gives us the highest abstraction but also flexibility. This problem is basically the classic problem in which driver you send to a customer. You can do this in different ways, but this decision is very important if you're processing hundreds of millions of orders. It impacts the customer experience and it impacts the driver's opportunities. It impacts the bottom line of the company depending on which driver you send. And this seems like an easy problem, but it is actually very important to do this correctly. But you should be involved in the systems here. On the bottom line, no. And so because we knew this was going to be an engineering system, I knew that we would need to have, we would need to get scarecatchers here. We decided to send it out here. And so what we did is we pulled the pretty basic system out of the store. This was a long time ago, but originally it was pretty simple. We just had an input pipeline, a training model, and after that we went to the race API and played as a group of business. And that was normal so at service we were sitting at the base just as a driver's and we made a difference from the base driver and then returned to the back end. And then I would be assigned to a customer and they would be able to kind of send customers here to this community and this nation. And we made the chance to bring it here because the size of it would be really awkward to actually know that this is a national environment. But we were feeling uncomfortable with this decision because the learning curve of Kubernetes was quite high compared to just employing a new cloud and some cloud emulation. So we were not sure about this decision, but over time as the systems developed and evolved we started to realize that it actually wasn't such a bad decision. So the next idea was we were deploying more and more models and services and what we realized we needed to have experimentation. So one of the great things about Kubernetes is it's standardized from control plan and API. This is something that a lot of people don't really talk about but when you're building a platform on top of Kubernetes, this is one of the biggest advantages to it. So what we wanted was just a simple interface for data scientists and other users to define traffic routing configuration to be able to UI for them and have that effect where experiments are run on the client for traffic in the cluster. So the driver list would come in where we already said we're routing preferences and the way that the request route is through a proxy. But this proxy gets configured by a traffic manager and the way that the traffic manager interacts with the configuration is through the Kubernetes API. So the API updates a config map and that templates a configuration within that router. So this would really well for us because there's loose coupling between these services and the Kubernetes API allows you to have these services loose coupling. So any service in the cluster that wants to subscribe or look at that data to perform its decision making can do so. So really this was paying dividends for standardizing on Kubernetes. The second thing we realized was that we wanted to support more complex types of models just having a model in an API is not good enough. So we wanted to be able to do something like have multiple models and then depending on the user and depending on the incoming request depending on the location for example send all of those models those requests and see which one is the most confident response or you send a request to two models. Maybe one is fast and one is slow but the accuracy differs. Then you have a timeout if one is too slow it takes too long but it gets back with a very good response when you use that. So the orchestrator that we bought for this was called NASA and you define these workflows. So this is just a very basic example with one simple step which is control step but you essentially have this long YAML that defines the workflow. So it's an inference graph essentially and a similar one you can define if you look at salons inference graphs which is an open source project. NASA is not an open source. Essentially what this logic to do is incoming request comes in and then orchestrates that request across these models and then essentially builds a hybrid or ensemble system out of multiple models. This is very important for us and already this was paying dividends because Kubernetes gave us a shared execution environment so all of these models are running in the same place. Standardizing that the execution environment was very important for us. We were calling an API and a VM and some other managed service. Everything is in one place and we could version control these workflows with our cluster configure manifest that consistency between those and the fact that you could look at it and say why don't you try to see what the whole cluster was doing was very compelling for us from an operational standpoint. And then there's some non-machine learning benefits to running ML as well. If you have your whole organization standardizing on Kubernetes then normal product engineers or infrastructure engineers can build tools that you benefit from and this is something that also a lot of people don't talk about but in our case we wanted to analyze experiments that were running. So the request comes into a model and some action is taken and we actually wanted to log those requests. Initially we logged into the console but we were experiencing back pressure from logging into the console because the throughput was so high. One of the teams called a logging the sidecar to run a container on your nodes and your log can just log to a port on a local host and it will publish those results to a stream and it will get sunk into a bit query where the data scientists can do analysis. So we benefited from logic engineers and local developers who were building these components, either packaged like as home charts or just containers that we could deploy in our system as well. So this helps us when we're building out our platform and for all these infrastructural reasons or all these areas we benefit from them. But one of the things I wasn't showing you earlier was that you don't just have model services in production. In real production systems you also have to have data available because these requests are coming in and they only need to enrich those IDs with featured data in order to make intelligent decisions in order to actually score and compare these drivers with a model. You can't just look at IDs. And so what we do is we need to somehow load data into these databases and there's two ways you can do that. It's either real-time or it's batch. And in the case of real-time, Kubernetes makes it easy because you can deploy order scaling consumers that just stream in data from PubSum or Kafka in our case and it fills up these stores. So in our case originally we just had Redis OX that we deployed with each model service. Those work really well for us. One of the great things about Kubernetes is that it has a jobs API. So you can do something like when you deploy if you trigger a job to put in a file, a 4K or a CSV populates a Redis that's been deployed. Once that's done the readiness prompts are good to go and your service comes online. So it has all this API functionality that makes focus training start-up and shutdown of these inviolable databases very simple. And one of the problems here was that I said one more thing to say next day. One of the other benefits we should do if you're using something like Kubeflow Pipelines and we're orchestrating large workloads, Kubernetes also has great resource utilization and schedule. So if you're training a, let's say Kubeflow and you need a GPU, it will find a node that has the GPU. If you need to have 100 gigabytes or 200 gigabytes of RAM, it will find that node. And these are small things I think to a data scientist, they just want to abstract it away but from an engineering perspective it can be hard to do that if you're using a VM or some other base abstracted platform. So one of the things I wasn't one of the things we ran into was that it was big problems was the duplication of our future data services. So in our case, a lot of these features were duplicated across these databases because they were independently deployed. Not just from a operational or production standpoint, but also from a creation standpoint. So we realized that we wanted to build a system that would allow us to deduplicate the work that data science were doing in creating this future data as well as provide consistency between our training environment and our serving environment. This project was one of our big projects which is a feature store and it essentially can be divided into two areas. You have two personas that interact with the system. On one side you have data scientists creating data in the form of batch data like datasets in BigQuery or files in an object store. They want to alter the data in the first place, but then you have users of the data, they manage use that are going to export and features a dataset from the query or some other store training model and they want to take that model into production. But we had a lot of problems with this. Firstly, there was a lot of duplication in the data that was being created here. Features of being recreated by teams over and over and over again. The death transformation calls this year will be duplicated when you go from batch data to stream data. So in batch data using Python in streams they're using Java and Go for performance reasons. This created a lot of model serving scheme for data production because the model being trained on data is different from serving. And it's very, very difficult to take. I think a lot of companies are tolerating some scheme there because there was a challenge between the challenging game and the consistency there. But also we had an issue in serving because we had to deploy so many various nodes. The operational challenge was maintaining these databases in production was very high. So we wanted to extract that away as much as possible because it's a very standardized pattern actually. So we bought a system called Feast which is a feature store which bridges these two worlds. This allows us to ingest in all of our batch data as well as our streaming there and store it into two stores. This ingestion job is consistent in that it's a single job that loads of this data and stores the same time. So you have a historical store of each data and BigQuery in our case but we could support other stores and we also have a real-time store which can be either big table or various. So these stores are consistent with each other because it's the same streamlining to both of them. So our annual engineer in this case has an API which will show you a little bit later but they can interact with this store to export data and they can train their model and then using that same contract of, let's say, the list of features they take the contract you're serving where they can query the data but on serving it's optimized for latency, so very, very low latency. Using real-time you can get 5 millisecond, 10 millisecond lookups sometimes even less but for the warehouse it's about scales so using BigQuery you can export terabytes of data pretty quick. So what Feast also provided was a way for us to centralize our feature management so everybody could find what features have already been developed and declared and registered so there's a nice duplication, more discovery more reuse and scalability. So if you go back to our previous slide so Feast in this stack would replace all of those Regist pots. So you'd have Feast serving that's constantly being updated by stream. All of your models can just with their specific contract of features that they need the feature data that they want on serving. So one of the questions we have is why is it important to have Kubernetes in this case? Why can't you deploy Feast on the VM or make it a managed service or something? I think the point here is that Kubernetes is very composable. Even if you have a complex system like Feast where you can deploy it in your cluster if you have a Pager at 2am that your system is going down you don't want to look at 10 different services and products you just go to your Kubernetes cluster and find everything there. All of your logs all of the data to debug that issue and this is what was compelling for us about Feast. And then our expansion. So one of the things that was also very challenging for us was firstly we had one of our products like GoRide and Lightning in Indonesia. We'd have an expansion of deployments and we'd run experiments across these deployments but we'd also branch out into multiple service sites right? So for each of those service sites we'd have all of these models running so the scale was exploding in terms of managing a lot of services and you don't want to hire 10 people to manage your 10xing of scale. We wanted to to technology solve the problem of scaling onto more service types and more markets. We were also expanding from Indonesia to Singapore to Japan and to Thailand. So the way we dealt with this was following a get off style. A get off style of version controlling all of our infrastructure as well as all of our Kubernetes manifest and code. So what this allows us to do is have a much better service to engineering ratio so we can maintain basically all of our Kubernetes clusters, all of the components deployed in there and then pretty much all of our code can get. So this is a very high portability when you go into a new market we can just redeploy that with some slight changes and some slight configuration differences. So this was very important for us but it's not just important in terms of scale human scale for example it's also important in traceability if you want to deploy something and you know that it's in a model it's coming a while and you'll go back and look at what actually went into that system. So I'll show you an example sorry there's also a great hint here about what I want you to use if you want to have a little bit of a get off technology. I'll show you why it's important to have that traceability. So I'll take you through a basic example of how we do CICB in the data science team. You typically have a model training pipeline that sets a huge load of pipelines. This will just take in some inputs and send us the features that say it's on data sets for whatever it is your target variable and you run a pipeline. So what we do is we track each one of these inputs, we track the pipeline code and then you'll have a bunch of artifacts that are created from that and these artifacts together will form a system and that system has some function that says it's your driver allocation system but it's important to know which of these components are going to make the system, the ingredients into the recipe and together you want to be able to version them. So what we do is we enter the artifact stores whether it's Git, whether it's a model store in our case we use MLflow or a container registry and we use Go CD so it's basically coding those registries for the latest versions and if anything changes they would say pipeline produces a new model it increments the version and employs that and so in this case it employs that as a new system into production and it assigns a new unique version to that whole system. The reason why this is important is because what you can do is you can track all of these ingredients into your or these components into your system and compare that to the outcome because this whole system is the thing that is actually influencing what the business outcome is and that's all you care about is the business outcome. So if it's doing really well then you can tie that back to specific components that went into that component and it's not doing well you can also understand why and this is something we're really focusing on right now standardizing this layer of tracking and tracing in GoJ. Obviously not just in this one system but in all of our systems and we're working with the Q-Play team to build a main stage layer that are doing most of the work we're just giving them our requirements basically and they're standardizing using a lot of open source technologies from the TFX team and gathering feedback from a lot of folks but internally we've got this layer but it's not standardized but we want to use whatever Q-Play team and community are presenting as best practices. So just to wrap up some of the good parts about Q-Play and ML on Q-Play there's a large ecosystem in the ML on Q-Play for machine learning it's really vibrant there's a lot going on right now so you can benefit from that but the standardize control plane is a great thing for building our platforms so Q-Play is not the main stage but building a platform on Q-Play is the main stage and it's good that running workloads is great as long as they're not stage 4 stage 4 is a little bit tricky having everything in a single environment makes operations a lot simpler and if you follow a GitOps based model or version controlling of your code it gives you a lot of portability and it allows you to track and trace what goes into your systems and monitor your outcomes and tie it back to those artifacts What still sucks is the multi-tenancy the biggest challenges is that data scientists are quite good engineers in a lot of cases they want to dive into the cluster and see why something went wrong or what's up with that and often you want to have abstraction there and you don't want to give them full access but you also do want to have them peel away the layers if they have to so the multi-tenancy aspect is something we're trying to solve right now stateful systems are hard so running databases that retain state within the cluster is a little tricky but a lot of managed services to do that extend to the cluster and then we still have a lot of leaky abstractions especially in Kubeflow where a data scientist is required to set an annotation or a CRD they said they want to access a GPU so that's still a flexibility concern and a trade-off but right now it is still a pain point so what's coming up next is we are looking at improving the user experience so we've addressed a lot of engineering and operational concerns with our platform we want to make it easier for that long tail of users and the data scientist is one of the easy UI and we want to address their concerns and give them a high abstraction and we're also looking at standardizing our metadata layer and then incorporating Istio for better tracing and for question and responses and I'll leave you on this tweet by Kelsey Heitar Kube is a great place to build a platform but it's not the end game and I think that's what we found as well building it out on Kube and Kubeflow so close to Chris I can take you through Hello I don't know if this is on This is working Sorry Oh, I'm going to use this one Okay, that's fine Yeah, there we go Can you guys hear me? Hi, I'm Chris Technical Program Manager at Google Clown so I work with people like Yan that you just saw and I'm helping Gojack with some of their ML platform challenges and I have the pleasure to show the demo today so I was one of the few product managers in Kubeflow in the beginning and Kubeflow as a project actually changed a lot over last year and there's like what Willem mentioned, there's a lot of activity excitement and a lot of contributors so hopefully we can see some of that as well So let me go through the slide here Kubeflow is an open Kubernetes native platform for ML and this is our mantra and this is our goal make it easy for everyone to develop, deploy, and manage portable, distributed ML on Kubernetes there's a lot of words there but those things are all kind of important and it kind of needs to work well together for a lot of data scientists for customers like Gojack to work well together and also let the data scientists focus on data science problem itself rather than the engineering or infrastructure problem Alright so I'm going to take a look at the demo here so we can kind of look at the Kubeflow and see what it looks like Alright so Kubeflow So this is our UI so there's a web app that you can use to deploy into your GCP project and we have a nice UI to kind of go through some of the applications so the first left here is the docs so this just takes us to the Kubeflow web page and then the first application here is notebooks also TF Job Dashboards so if you're to deploy a TensorFlow job monitor how it's actually working and then here's Catib which is our hyperparameter tuning application that we work with in Kubeflow and then Pipeline Dashboards so I'm going to go through what the actual pipeline looks like once we launch some of those experiments but let's start with notebooks since that's where everyone works in the data science world so if you were to deploy a new server or spawn a new notebook this is what it looks like for us so you can just easily deploy a new notebook server by filling out a form here as we provide a lot of standard images that you can use to quickly start work for data scientists but also you can specify a custom image so that you can put your own requirements in there for enterprise customers that require special packages and libraries and whatnot and you can specify some of the resources that you require so like what Willem said earlier for data science it's important to focus on the work that they're doing rather than worrying about CPUs or memories or things like that so Kubernetes do a lot of that work for us and Kubeflow abstracts it even more by making it easy for data scientists to work in the workspace here you can attach a space volume like hardware sorry not hardware like a disk if you need to add more data for the work and specify any other extra resources like GPU over here and then spawn the notebook server so I've already done that for us so let's go back cancel maybe and then in the demo and if you were to connect just what our server looks like and I took out what the demo notebook over here that one this guy over here there you go so I'm going to go through what's in the notebook and in the in the process of explaining I'm going to turn it back over to Willem to talk about fees and how that really looks like in kind of an example workload here so in this notebook we're working with a data set that is collecting all the taxi ride information a fare information from Chicago and this is a popular data set for data scientists to do any research or practice any of their models and to predict some fares so in the beginning of the notebook we are defining some requirements and importing libraries let's go down so as Willem mentioned here makes it easy for data scientists to define all the code and then put it in a notebook and not leave the notebook and be able to manage all the work that they need to do to package it up and deploy the work so at this point if you're a data scientist working with the data set you probably want to take a look at data load all the libraries and then let's see what it could potentially look like so we're going to use Feast to load the data but you could potentially use a streaming logs into Feast but right now we're just using Pandas data frame here to just illustrate that we're going to load the data into Feast so in defining the features of the data frame or working with some of the features here around distance pick up latitude some of the locations and also not only the origination but also destination where things are going where the passenger is going and a couple of the other features that might be important so we're just trying to see maybe let's look at some of these features see what it could potentially look like and at this point we're going to use the Feast to load the features in the data for addition analysis so let me turn over to Willem right so at this point we're instantiating an importer and what this importer is doing is essentially taking a Pandas data frame and it's starting a data frame job in this case it's data frame but it doesn't have to run a data frame it can also be a beam direct runner if you're on frame and it loads that data frame into Feast so in a production environment you would have this step at the back of your ETL pipeline the final step where you transform your data or you'd run a similar importer stream to stream so you wouldn't have to do this every time you'd just be consuming data from Feast but in this case we're just illustrating this for the sake of completeness so what we're doing here is we're ingesting that data frame and we're also defining the features so we're mapping the columns of the frame two specific features and if those features do not exist yet we're saying create them with this apply feature and apply entity so it's going to create a write entity and it's going to create a list of features and map the columns of the data to those so we've written around this import so you can see it's creating all those entities and features and then it's running an import then what we do is we are defining a feature set and this is one of the key things of Feast is having essentially a contract between training and serving and that contract in this case is the feature set that you're going to use to train a model so in this case the two lists here are the entity and the training feature set and we create an instance of this and then what we do is we retrieve a data set and materialize that data set as a data frame locally so in this case it seems a little bit redundant because we've just ingested and extracted again but in production the data will come from many different sources so we've extracted that data and we've done it over a time range and if you run this hate you'll see that it is actually a data frame and the data is available there so I'll pause it back to Chris and let him continue with the demo Thank you So as Willem said we have the data in Feast and if you're a data scientist before you work with data you probably go through an exploratory data analysis phase where you kind of want to take a look at and take a peek at the data get some statistics and try to see if there's any predictor that might be helpful for your data science problem So in this case we're using an application called TensorFlow Data Validation or TFDV and we get a lot of questions like how does TFX ecosystem work with Qflow so internally they work really closely together to make Qflow the best place to run TensorFlow extended components like TFDV and Qflow makes it easy for TFX document just to run because everything is managed on Kubernetes So if you were to run TFDV over here and run the Visualized Statistics Command and then you get a nice UI with a bunch of important statistics values like mean, standard deviation and other information about the data that could be useful for data scientists and this visualization is done through facets and you can actually interact with it It's kind of nice to see and get a little more information around it and maybe get some good idea about maybe I can work with this data or not or if you want to continue to munch it to see if it makes sense then you can continue working those notebook cells I'm kind of happy with the feature set that I have it looks like it has all the things I need then I'm going to create this feature set and then initiate it and also write my training code so that I can do my linear model linear model prediction in this training code if you look here that we're going through the training feature set from Feast that was defined earlier and just going through them and then creating our training code for training the model and in this part right here we're specifying the time range so that if you were to run some different types of predictions or training sets based on time range maybe that's a good way to work with the data for your problem in this prediction code over here we're logging some information and also working with different types of features for prediction workload and as you can see we're trying to minimize the serving and training skew that we were talking about before by using Feast and using the same column names and there's some saving the model and we run the training over here and you're using the same image and the same model and run local prediction and maybe at this point you might be happy with the prediction value that you get and it might be okay with it and now you're ready to perhaps deploy the model so instead of taking this model and maybe going through different application or product to take the model and deploy it you can do everything within this one environment notebook because of the fairing so fairing is a project within Kubernetes to help the data scientists by packaging the code that they wrote from notebook and turn it into a container and deploy it in Kubernetes so in order for you to do that there's only a couple things you have to specify so there's two to put in here or there's a base image in the location of where you want to publish it and you can do a little more by just writing some builder code to build it and then all you have to do is launch the job with a couple more code and then you're done so at this point what it would do is as you can see from the logs it would just launch the job to train the model and try to see if you can do more of a distributed training type of workload here let me just flip down here okay so after the training is done that we have the trained model that we can deploy the model using Kubeflow with a couple lines over here and sorry and with the fairing one of the benefits that it can have is that you can easily make a endpoint here with two lines of code over here so you can do a rest of the API call through that endpoint and you can with a couple more lines of code you can call the prediction endpoint and see how it performs and perhaps you are ready to deploy this in production sorry deploy this in production using to be able to repeat this in pipeline so Kubeflow Pipelines is a project that was developed by Google to easily allow the steps of work that you have seen so far the data scientists would do a notebook and package it and be able to deploy this pipeline and also make it reproducible and be able to also share with the other folks within your enterprise in Kubeflow Pipelines SDK there's a couple lines here that you specify the name and the description in the pipeline and you can point to the model and each step of the code that you saw earlier the training, the prediction and deployment the validation deployment they can be dockerized containerized over here and it will become an operator or it will become an operation within pipelines so you just specify for each of the steps give it some name and also what kind of range of data that you want like you saw earlier and package it up the validation step and with couple more lines you can deploy it into production and by submitting to the pipeline run so when you submit the pipeline it will look something like this so that it will give you a nice graph of how each of the operations look like in succession and this can get really complicated depending on what you're trying to do or it can be very simple something like this so let me go back to the presentation here so what you just saw was that we loaded and also fetched data from Feast for training and this is a very positive thing for data science because again like what William said you do not have to necessarily recreate features all the time and duplicate it so you just take on more storage or cost more for the company or it just takes longer so when you publish those features into Feast you can also minimize the training and serving skew or eliminate it from the same features from same feature store so as you saw that we took those features out and also developed a model through the training code and we deployed those trained models into Kubernetes by doing the deployment and fetched data and serving time for inference from machine learning problem so in QtFlow in our project we want to be able to provide all these steps that are typical for machine learning development in one platform we want to make it easy for data scientists to do their job and not have to worry about the infrastructure challenges and any other complexities that they might not be so keen to do like what William said most data scientists are strong engineers but then they don't become data scientists to do engineering necessarily but they want to do more scientists type of work so in QtFlow we want to make it easy to do their job to do their data scientists work and also want to give them option to switch out different components if they want to so for example in serving if you're doing developing model intensive flow you can use the key of serving or if you're using PyTorch or XGBoost you can use Selden and it's very easy to switch that out and then just make it part of your deployment story QtFlow makes it easy for anyone to run these steps on Kubernetes and here are a couple of tenets that I want to just highlight composability, scalability and portability is important for QtFlow project so because we know that your first development workflow typically starts within your local disk or sorry local laptop so we want to make a data scientist workflow not very intrusive so we want to make QtFlow easier to use in a local environment or even in on-prem a couple of the customers that I work with have their own data centers or clusters that with specific hardware requirements or regulation issues so we want to be able to use QtFlow in any environment and obviously as you saw in public cloud in Google Cloud you want to make QtFlow easy to use there as well in addition like the number of users and workload size these things should not be an issue in my personal opinion for data scientists to worry about it just needs to work it just needs to be able to burst into cloud and scale out and composability using your own library or framework like Feast because Feast is a different open source project that was also brought in to be part of the QtFlow ecosystem so it's quite difficult to include other folks that want to be part of the QtFlow platform and this is proven by a lot of other components that you can see when you go to our website and then see all the other different contributors quickly in the QtFlow architecture because I see the time is running out this is what our QtFlow cluster architecture looks like I'll just give you a second to look at that are you guys going to show through this towards the slides? Is that right? Yes. I think so Yeah So when you deploy QtFlow everything is within one cluster and then you can see the shared namespaces which makes it really easy for people to manage their workload and also you can connect to either the laptop or the cloud resources and have those data sets either available in cloud or you just want to download some of it and then use from your local disk for whatever reason for exploration reasons makes it easy for us to do that As you can see there are many companies that are part of the QtFlow project that we have over 30 companies or so that not everyone is listed here but there are some big names that you might recognize and from the number of PRs 28 days you can see that it's exploding and this actually just keeps going up and you can see this is the end of April so it just goes until There's a lot of momentum and I see new working groups that are also being performed so originally there was only one working group but after I joined we started the product manager group and now there's a serving group and an on-prem group with the launch of Anthos I see a lot of clients that are interested in using Anthos and QtFlow on top of that so that they can use the Kubernetes that are on-prem and also in public cloud as a seamless experience One of the goals that we have in our project is low bar, high ceiling so we want to make it easy for people to start but also expand and make it as complex as they want to because not all one size fits all we want to make it easy for everyone to bring their own requirements and use our platform to solve machine learning challenges Just a call to action slide here I would highly encourage you guys to check out our website and install QtFlow and also install Feast Feast I think is one of the key missing pieces that we had Feast is super useful for any production data scientists team and also give it a shot trying fairing plus feast a notebook that you just saw so that's available in this link Well, thank you very much for your time and then hope you guys have a rest of the day Maybe questions? How much time does it take for you to put this together from the moment of that?