 Hello, and thanks, everyone. Thanks for being here. And for those of you who are watching live, good morning, good evening. Good afternoon, depending on which part of the globe you are in. We are at the open source summit, North America here in Seattle, and today we are going to talk about machine learning pipelines and lineage, right? And specifically on top of Kubernetes, right? We all know Kubernetes as the container orchestration platform, but off late, you know, there has been a lot of great work happening in terms of establishing machine learning and ML Ops platform on top of Kubernetes, and this talk is going to go into some of the aspects of it. Myself is animationing. I'm the CTO for Watson AI and Data Open Technology, and leading a lot of engagements in open source from IBM. With me I have my colleague Tommy Lee. Tommy, you want to introduce yourself? Sure. Yes. Tommy is part of the team and leading a lot of the initiatives here, so let's move on. Now, one of the things which we are noticing definitely is, you know, enterprises are still struggling to scale AI beyond experimentation. I mean, a lot of times the news come that, you know, I've developed my model. It has more than 11 months, the model is still not deployed in production. Even if it is deployed, it's hard to measure the business impact of those deployed models. In some cases, right, there is not trust. The organizations or the end users who are supposed to take action on these deployed models, they don't have trust in these deployed models. So now, one of the things is like, you know, the best practices for building accurate models are well understood, like how do you go through different parts of the life cycle in terms of producing these models. What is not is essentially, you know, how to create these production data science platforms at scale. Right, you need a holistic architecture. You need model management. You need integration with the rest of the enterprise silo systems. You need logging tracing. Then, you know, you need more effective engineering, which means, you know, a lot of automation, a lot of standards and a lot of pipelines, right? Unless you standardize and automate this whole field, you're not going to be able to scale to the extent you desire. And last but not the least, you need smooth operations, right? It's one thing to deploy. It's one thing to monitor and manage. The day two, which is much harder, right? So you need a lot of monitoring around models, around your maintenance strategy, et cetera. So I think we understand clearly that, you know, the three pillars of AI life cycle, right? We have data sets. You use data to build models, which then automate decisions, right? So data and models are the two pillars. And when we look at what is the third one, that essentially is the pipelines, right? You cannot do this breadth-oriented, very manual intensive procedure unless you automate and the way to automate is using pipelines, right? Starting from data cleansing, feature engineering, data ingestion, all the different fields you go through. And then when you come to the machine learning part of it, when you're running distributed training, you're running hyperparameter optimization, then you are running validation on top of your trained models, finally deploying them. But then, you know, your journey is not stopping there while deploying, right? You need to be monitoring for drift and normally other things, right? So all these things need to be automated. And the way to automate this end-to-end life cycle is through pipelines. You cannot, you know, overstate the relevance of pipelines in this whole ecosystem to actually scale and produce, you know, AI models in production at a scale which you desire. So for that, one of the projects which became very popular in the industry, Open Source Project is called Q-Flow Pipelines. Essentially, you know, it's, think of it as a CI CD pipeline as a very basic level, something which can run on Kubernetes. And then, you know, when you run on Kubernetes, each of your tasks can be encapsulated inside containers. At the basic level, that's what it is, but then you start working towards, you know, machine learning capabilities. What are the machine learning capabilities in it? So essentially, you know, there is a Python DSL so that data scientists can actually program this whole pipeline end-to-end using Python directly. No need to deal with Kubernetes, YAML files, dog and containers. If you just want to use Python, program pipelines, launch it, use it, right? The other part is, you know, there is a lot of capabilities built around lineage and metadata which I will, you know, go into some of these details which are very, very important in the machine learning world. So for example, in this case, like when you see this definition here, this is like a 15 lines of Python code. What it is doing is, you know, it's doing TensorFlow data validation, doing preprocessing, then, you know, training, running distributed training, using TensorFlow under the covers. Then finally, you know, once your model is produced after training, it's running model analysis and then deploying your model in production. All end-to-end data and machine learning lifecycle done through these 15 lines of code. Now, obviously, somebody has before that, you know, provided automated, you know, templates for these to be executed, but here now our data scientists can come, program these pipelines and run them, you know, repeatedly at scale. That's the power which Kubeflow pipeline brings using the Python SDK and makes it very friendly for data scientists. Now, when we decided to productise this within Watson and within IBM, one of the things which we did, so Kubeflow pipeline by default, the architecture essentially is that, you know, it uses Argo, which is a CNCF project, right, as the underlying container orchestration CICD engine. We replaced that whole thing and re-rode the Kubeflow pipelines to actually run on top of Tecton. Now, Tecton is another project. It's part of the CD Foundation under Linux Foundation. Initially started by Google, IBM and Red Hat are prolific contributors and we have standardised our whole cloud pipeline story around Tecton. That means Red Hat OpenShift pipelines, they are based on Tecton. IBM cloud pipelines, they are based on Tecton, so it made all the sense for us to build our machine learning and AI strategy and the pipelines on top of, you know, Tecton. So we leveraged, you know, the Tecton substrate under the covers and re-rode Kubeflow pipelines to run on top of it. And now, you know, you get all the machine learning capabilities from Kubeflow but running on top of Tecton, right. Now, if you look at Tecton at a very basic level, essentially, you know, it exposes a bunch of Kubernetes custom resources for tasks, pipeline runs, task runs. I wouldn't go into the details of it, but it suffices to say think of it like, you know, task is your unit of work and task is essentially made up of different steps, right. And each task is essentially, you know, running in its own pod and the combination of all these makes up a pipeline, right. And that's how very modally architected Tecton is. So now let's talk a bit about machine learning capabilities, right. What are the machine learning capabilities beyond the Python DSL? Is it just the Python DSL? No, there is a lot of metadata logging which Kubeflow pipeline provides. Now, why do you need metadata logging, right. Hey, first of all, you need to find out, right, trace back what dataset my model was trained on, what hyperparameter was used, what TensorFlow version was used, because tomorrow if your model is making decisions like denying someone a loan or admitting someone to a university, you will need to have that auditability and tracing, right. And metadata logging provides you that, right. So you can trace back all the way to the data to the hyperparameters to the framework version. And then you can compare various different experiments which you are running, right, to train your models, which one is performing better on different parameters or which one is running faster, right. Not only this, when you are looking at the field of transfer learning, where you get actually some base models and you need to produce more customized industry-specific models, metadata logging can help you in that as well. And last but not the least, right, in terms of being able to cache. So if you're running these very, very long pipelines which are running distributed training, and you probably, you know, at some step, 30 minutes or two hours down the line there is a mistake, you don't want to go back all the way and start again. You want to be able to cache and move forward, right, in the next run. So metadata logging also provides that capability. Then there is artifact tracking, right. So as I was talking about the artifact tracking, essentially here is all the artifacts which are produced, whether it's the next version of the dataset or the model which is produced. All these things are logged as different artifacts under Q-Flow pipelines and you can go in their all version nicely and you can look at the history. And then combine this, all these different artifact tracking and the pipeline runs and you get linear tracking which essentially allows you to go all the way back from your produced model. If you can see the model is here but you can trace back all the way from the model. How was the dataset transformed? What was the initial dataset? So you can trace back everything all the way to the back using this linear tracking capability. And to enable this metadata and linear tracking, one of the things which Q-Flow pipeline on Tecton uses is essentially what is called metadata writer which is watching all these different pods which are producing these different outputs onto these empty directory volumes and then it's watching and copying them back to machine learning MLMD which is an open source metadata tracking system which was launched by Google. So this is the solution which helps us track all the metadata as part of the Q-Flow pipelines. And once it is running, it's all seamless. What you see is as the pipelines are running, you see all the logs getting produced in real time. If anything is going right or wrong, you can see all this and then you can look at the MLMD data from this tab and you can launch the different artifacts and from those artifacts then you can do the linear tracing where you can trace back the lineage from the model deployed from the scoring endpoint all the way back to the data on which the model was trained. Now this is all open source, right? And on top of this open source, we launched our product called Watson Studio Pipelines. So if you're not aware of Watson, Watson is the IBM's machine learning and AI umbrella. All the platform products which we launch under the AI space come under the Watson branding. So Watson Studio Pipelines essentially was just launched in open beta and it's part of this ecosystem of MLops portfolio of offerings which we are launching under the Watson brand. And that essentially gives you a lot more capabilities on top of that open source solution, right? Obviously provides the capabilities which are there, but then gives you a drag and drop interface, right? So you can drag and drop different components and construct your pipelines if you don't want to write Python, right? For example, so this skaters also towards business users. And then there are components, right? For all the different IBM products like, you know, data refinery, which is our tool for data manipulation, AutoAI which launches AutoAI experiments. So a lot of these pre-built components are provided for you to be able to drag and drop onto this palette, construct a nice pipeline and launch it. Now as you can see on this particular screen, right? This one is using AutoAI. It's using DataStage. Now DataStage is an IBM product which you'll be surprised it has been deployed at more than 2,000 plus customers for the last two decades. It's very, very popular product in the ETL space. Now with the power of Q-Flow Pipelines and Watson Studio Pipelines, you can just drag and drop that particular product capability onto this canvas and string it with other capabilities from Watson and Breva. Okay, so what did we talk about so far, right? We talked about that we need pipelines to heavily automate machine learning in AI and move machine learning models in production. For that, the project which we talked about is Q-Flow Pipelines and we have a special flavor of Q-Flow Pipelines which is built on top of Tecton. Q-Flow Pipelines on Tecton is our machine learning and AI pipeline engine. That's an open source and on top of that we have launched Watson Studio Pipelines which is the product version. Now there is one more thing, right, which still doesn't exist, right? What doesn't exist? So one of the things which we are seeing across enterprises is that there are different groups within companies which are getting formed which are focused on different parts of the AI lifecycle, right? Some teams are doing feature engineering, some teams are producing data sets and different versions of the data sets. Some teams are producing models and then there are teams which are producing pipelines and individual steps of these pipelines. What is happening is a lot of duplication. Sometimes an organization within the same company, even smaller companies, they are producing very similar versions of the data sets which they have modified or very similar versions of the models. Obviously a lot of the pipelines and the pipeline steps we are seeing, you know, getting duplicated a lot, right? So there is a need for this one central place where you can actually essentially go share what you are building within your organization so that it can be shared across team and organizational boundaries, not within your organization only, right? It can probably be shared across, right? And then the other part is, you know, when you are picking up, for example, you want to pick up a data set or a model from the public repositories, you want to be assured of the licensing, right? You don't want to pick up something and train your model on a particular data set where some and some ensues you, hey, this is not the right data set or you don't know the lineage of that data set, where did it originate? What was the licensing? What is the auditability trail? So you want to be able to go to that one single certified place where you can, you know, download assets where you have the right set of metadata and lineage. Now to handle this problem, we are joining hands with Linux Foundation AI and Data and we are announcing Machine Learning Exchange. So Machine Learning Exchange essentially is built around the concept of sharing and reusing pipelines, datasets, models, notebooks, pipeline components and this is a project which we are launching jointly with Linux Foundation AI and Data on their infrastructure and we are hoping, you know, more and more partners will join on this mission, right? So essentially, as I mentioned, right, you can have, this is one single central place to go. You can hit ml-exchange.org right now, go there and you will see, you know, certified datasets all with, you know, CDLA license, pipelines, notebooks and models, which essentially, you know, you can start downloading and using, right, with the right provenance. Now, under the covers, why is it relevant in a pipeline's talk? Okay, because under the covers, you know, the components which it is using apart from, you know, sharing a lot of the Kubeflow pipelines and components, it's using the Kubeflow pipelines on Tecton engine for executing all these artifacts. So it's not only an engine where you can go share, download or upload, you can essentially also execute. So if you deploy your own version of machine learning exchange on your Kubernetes cluster, then, you know, you can upload your own assets and you can actually launch them. So the pipelines which are there, hey, you should be able to try it out. Yes, you will be able to launch. The models which are there, you should be able to deploy it. Yes, you would be able to deploy it right there on that particular engine, right? So that's the inbuilt and the integrated capability which machine learning exchange comes with, with Kubeflow pipelines on Tecton as a pipeline technology as well as, you know, another project called K-Serving or K-Serving, right, for serving your models in production, right? So it's pre-integrated and pre-built into this. So as I mentioned, like one single place to go, look at your pipelines. If you're running your own version, you can register, search for pipelines or you can, or upload your own pipelines. And not only that, you should be able to launch your own pipelines as well, right, with the pre-integrated engine in this. So as you can see, when you click the launch button, it comes with all the different input parameters which you can enter and run it. So once it launches, as you can see, you know, it's using the underlying Kubeflow pipelines on Tecton engine. Similarly with components, right? So components themselves are the individual building blocks of the pipelines, right? This gives you one central place where you can upload and share all the components and you can try them out before you integrate them in a pipeline, right? A component can be as simple as, hey, echo a hello world, right? But then, you know, once it is there, nobody else in your organization should be creating echo hello world, right? I mean, that's the idea. And then you should be able to test it out as well before you start using it as part of the pipelines. Models, definitely, right? They are at the heart. And at the center of this, you can go and a lot of the models of a system project called model asset exchange. A lot of the models initially come from there. We are also integrating a lot of the ecosystem models, but models around, you know, object detection, text sentiment classifier, you name it, they're all present there and you should be able to download and use them, right? And if you're running your own version, you should be able to upload your own models and register and expose. And datasets, right? One of, you know, the integration of datasets within the industry and they are at the very beginning of the machine learning life cycle. You need different datasets to be able to, you know, train and create, you know, more refined models so you can actually go look at a dataset, look at the metadata surrounding the dataset, the licensing and then, you know, you can essentially use the pre-built capability within machine learning exchange to download that dataset onto your Kubernetes cluster or OpenShift cluster or, you know, you can take that dataset so now, so that's the machine learning exchange framework and in the catalog as I was talking about, you know, there is a lot of high quality curated content, right? And just to give you a sample of the kind of content which we have in machine learning exchange, for example, you know, one of the datasets which we have, it's the data for, you know, code, essentially think of it as it's, you know, AI for codes, ImageNet equivalent, right? Which allows you to do different kinds of code analysis, it allows you to create code, it gives you suggestions how to make your code performance. So a huge dataset, in fact the largest AI for code open source dataset which is right now available extends as part of the machine learning exchange, right? And if you want to see the breadth of it, it's like 14 million code samples with around half a billion lines of code, right? That's the amount of code which is available in the AI for code dataset and this particular dataset is available as part of machine learning exchange, right? And not only this dataset, right? You have this dataset but you also have, you know, subset of these datasets for a specific task because as we talked about, right, the dataset is humongous. You also have associated notebooks, right? For example, in this case, you see a notebook here for project coordinate language classification, right? These notebooks essentially allow you to take that largest AI for code dataset and then, you know, create more data. So, all in single place, all these related assets. Please give it a try, ml-exchange.org, very easy to hit and once you go there, it will redirect you, I mean not redirect but you have a link there to the GitHub as well. Please go to the GitHub repository, try it out and give us your feedback. Okay, with that, I'll pass on to Tommy to actually talk about some of the advanced Qflow pipelines and Tecton capability and also show you some demos. Tommy? Hi, I'm Tommy. So now I'm going to talk about some of the advanced Qflow pipeline and Tecton capability. So, as we first implement Qflow pipeline on top of the Tecton engine, we realize a lot of the advanced Qflow pipeline features such as parallel loops and recursion cannot be done with the native Tecton capability right now. So, this is why we talk to the Tecton communities and we have come up with this new concept called Tecton Custom Task. So, this is a Tecton feature that you could use to plug in alternative execution engine into existing Tecton pipelines. And they are actually processed in a separate controller. So, the user could actually define their own weekend style logic that's much differently than the regular Tecton task. Just for some background regular Tecton task is created with like a new part to run this code. So, it's actually always containerized whereas in Custom Task, it's actually inside the same cluster. And further forward, you could also add more complex execution flow such as loop and recursion on top of your weekend style logic. And in Qflow pipeline with Tecton, we actually make Custom Task a plugable pipeline component just as other Qflow pipeline components you could use in the Python DSL. So, now I want to go over some of the feature timeline within Tecton Custom Task it's been a long journey on this feature in Tecton. So, when we actually first talked to the community, they introduced this Custom Task feature back in June 2020. That's when they asked the proposal and introduced the new run API in back in July 2020. However, at the time like the run API cannot be run inside a Tecton pipeline API. So, this is why our IBM team kind of stepped in and try to contribute and make it closer as like the regular Tecton Task API. So, we introduced the Custom Task API inside the pipelines back in December 2020 and add like additional capability such as like supporting workspace service account part templates early this year and Custom Task result in February 2021. And recently in the past few months, we also like try to fill in like more gaps in like the TaskSpec feature. So, we actually introduced like embedded TaskSpec for Custom Task back in June. And also, we have supported Timeout API just last month. And at the time, we actually try to fulfill like the rest of the API gap from Custom Task at the moment with the communities. And we have the proposal accepted as well last month. So, now I want to introduce one of the main Custom Task were created for QPOP pipeline in Tecton to fulfill some of the core missing feature in QPOP pipelines. So, one of the core feature we want to fulfill in QPOP pipeline is able to loop and recursively run a same subset of the pipeline over and over. So, this is why we introduced this new Custom Task in Tecton called pipeline runs where the main feature is able to like typically loop over a set of parameters. And additionally, it could also loop into multiple arguments inside a list of like JSON requirements as well. And last we also handle like conditions so it could actually run loops recursionally. And in this Custom Task, our main goal is first be able to like support loops on a subset of pipelines so it could handle like the QPOP pipeline in Tecton DSL such as parallel loops. And second of all, we want to able to create sub pipelines which is equivalent to QPOP pipelines graph components in this case. And the third thing, we want to support a recursive reference where we could actually create like wild loops using like recursion loop inside DSL. And we also have support for parallelism which is also like a feature existing in the QPOP pipelines as well where you could limit the number of parallel blending graph within the same parallel loops. And last, we also have the capabilities to optimize like some of the retry and timeout strategies. So let's say if one of the parameter in the loop is failed, you don't have to retry every parameter in the loop over when you are able to optimize retry strategies. And furthermore more than the looping components, we actually have introduced all the Tecton Custom Task to optimize our workloads in QPOP pipeline as well. And one of the example we have introduced is Custom Expression Language which you could actually process much complex condition just inside the controller without creating any new parts. So you actually optimize a lot for the container run times. And then another example is able to for commonly used tasks such as just sending notifications could be also just done inside the controller without having created new tasks for every pipeline when you just want to send a notification to one of the user, one of your organizations. And then lastly we also introduced some of the new components in QPOP Tecton. One of the components we have introduced is called any sequencer. So any sequencer in short is a dependent task that actually starts when any of the tasks or condition dependency completely successfully. So that pilot actually doesn't have to wait for all the task dependence to be complete before moving to the next steps. And this can also be done in some conditions. So let's say in this case you have Job1, Job2, and Job3. When Job1 completes any sequencer could actually just trigger Job4 directly. Or when you have Job3 complete you could actually verify if Job3 complete in a certain way before you trigger the condition for running Job4. So it's actually a way flexible dependencies to run on QP in Decton. So let's look at the demos on how this actually works in action right now. So I will go over the demo for running the any sequencer on key flows pipelines. So let me bring up the sample over here. So we have uploaded a pipeline that is running any sequencer. So any sequencer in the base form is just a regular key for pipeline components where you could actually just depend on three other different tasks. And then when one of those tasks actually completes, then the any test sequencer task could actually just see it and actually trigger the next components. So let me start to run these examples so you can see in actions. Yes. Oh, sorry, yeah. Give me one second. Thank you very much. Yes, so we just execute like the any sequencer pipeline. So let's take a look. So to begin with, the any sequencer pipeline will have three components that these are just sleep components as you can see. They actually sleep for different section of the time. So one of them actually sleep for 15 seconds, another one sleep for 200, and this one sleep for 300. So this is a task called flip coin. So we flip a coin to say whether the result is head or tails. So let's take a look at the result of this. So because this actually says it's tails, so the any test component I just see is not expected because the any test component expect a head condition before it actually trigger the next drops. So we actually look in the logs for the any sequencer. You could see, okay, the flip coin however the condition didn't met as they expect the outcome. So it actually not going to start the next components. It will just continue to wait for either the sleep component to a component free to be complete in order to proceed to the next task. I also have a previous run where it actually returns head. So let me go on this one. Oh sorry, actually this one. So in this one because the component actually returns head as input. So you could see it. So that's why it says the task run of flip coin succeeds and continues the next task where it runs the drop for sleep components. So in short, this is how like any sequencer work in actions. Because so now we're going to like talk about some of the other demo we have prepared. One of them is actually what animation have mentioned about. We have a managed service on IBM cloud to run give a pipeline in tectons. So what we have called is the Watson Studio Pipelines. So for Watson Studio Pipelines you could actually create this on any IBM cloud Watson Studio service. So when you go to Watson Studio projects you could click on add the projects. We will see like a pipeline managed service. When you add this and you actually will see one of the pipeline work with it here for the demo. This is what we call AutoAI Pipeline to actually pool like data sets and automatically turn the hyper parameters and then find the best result of the pipelines and deploy them on Watson Studio so you could actually use it in actions. So once you get into the Watson Studio Pipelines you will see it's a little bit different from DQ4 pipeline dashboard where in Watson Studio Pipeline you actually have the ability to drag and drop different components as you can see over here and like create new dependencies and then you could also simply just remove them if you don't want to and when you double click it you can always modify the different input output parameters and one of the cool feature we found is like having the ability to able to add annotation and comments actually helps you understand what these components actually try to do like when they see in the UX of a dashboard. So this is what we find very helpful and once we like prepare the pipeline well with all the parameters then we could actually click on run and start running these pipelines to select the datasets and then we will select a Watson Studio space to deploy our models so at the end we could actually go on the Watson Studio space to run the predictions and then we also fill an API key for your relevant services that you use in these pipelines. So once we click on runs behind the scene you create like a pipeline for you on a Kubernetes cluster and then we will run this step by step just like the key for pipelines for open source. So because this pipeline will take like several minutes to actually like get the dataset and train the models so I will just show you what the problem we have previously run. So in this case you will see all the pipelines has been complete and you can also check the logs once the pipeline is finished and you can also click on here with the pipeline outputs where you can see all the output for each components and on one page wheels which is very nice. You can actually keep track of it without actually click on each component and once the pipeline is complete you can still like add the annotation part whereas if this pipeline is very good let's say modify the annotation a little bit just let the datasets know what is going wrong and when the next datasets take a look at this pipeline they have some idea how to keep track of stuff for each component inside this pipeline. So this is an overview of what was the pipeline looks like and last and not least we are going to just show you a brief view of the new open source project we will introduce called machine learning exchange. So in this case this machine learning exchange is actually running one of our clusters that could execute pipelines on this. So actually let me enlarge so people could see it better and in this case we have several different kind of assets so one of them is datasets as Anna mentioned one of the cool dataset we have is called project code net where we actually have 40 million code samples and we click on this dataset it actually shows you all the data set what kind of license you have used and you want to just have a more short view of the dataset metadata itself you could see in the YAMLs and of course you could also launch this dataset on the same cluster so it actually create a volume and provide a dataset that is ready to use for your other assets such as pipeline and notebooks. So once the dataset is ready you actually could run a notebook to use this dataset so let's say we pick a dataset for the project code net let's say for language classification this notebook uses the subset of the code net dataset where of course you could actually launch it directly over here but you could also preview this notebook code to see what this notebook is actually trying to do so this notebook will first just try to import a list of libraries to process the dataset and then this is a classification model so you could see this is the list of classes we're going to classify so because we're only running a subset of the code net so we only have this 10 different classification class in this model so because we don't have that much time to run over this whole notebook we actually pre-train the models beforehand and register as a model asset already so in this case we pre-train the language classification models on machine learning exchange and register as a model assets over here so inside the model assets you could also provide a readme to describe how this model is being trained what kind of data is being used and of course you could also generate samples in the background or run on your own Q4 pipeline instance so because it actually integrates with the Q4 pipeline we have built inside machine learning exchange so we're actually going to just launch it directly over here so in this case we're just going to launch this model inside a Kubernetes cluster with a simple deployment and services to launch it is very simple so you just have to provide what kind of action we're going to do in this case just create a platform and we'll pre-define the running name for you so you just simply click on submit behind the scene we will create a Q4 pipeline with the integrated platforms and you could see they actually create a pipeline behind the scene first extract the config from the machine learning exchange model and translate it into something that you could understand and once you see this is done it will create a separate component that actually take the deployment from Kubernetes and actually deploy on the same cluster so the user can actually navigate and try out these models so I believe the Wi-Fi is a little bit slow but once the model is complete you should be able to see inside the logs the model is being deployed and this is like the endpoint that this model will host it could just directly click on this endpoint and because this model is already wrapped up with a swagger API so you could actually just directly using a simple post with the built-in swagger and in this case we will just do a simple model prediction so we need to actually try out these models you could just upload any coding files I will just pick a Python file let's say in this case and then we just execute then you could see okay this deploy the models actually give me a result okay this file is written in Python with 0.73 confident rate of course this is a training with a subset of the code data sets so that's why the confident rate might not be that appealing but of course as you train with more models and more sophisticated algorithm like this confident rate could be improved and more suited for your use case and this is the end of our demos thank you very much and please check out our other talks we have over the open source summit as well and one of the talk we want to highlight is tomorrow morning we have the keynote with Tom Moore our EP's to talk about more in details on how advanced data AI quantum using open source and we will also introduce versioning exchange in more detail over there thank you very much for watching is there any question online or in person is there any question in person or do you see any question online just one there yes go ahead so oh yeah yeah yeah so for the tacton custom test the question is about is the custom test is outside of Q-Flow right so right now these two tacton custom tests we actually bring in as public Q-Flow problem in tacton one is the problem which is actually integrated as part of it another one is the common expression language that's also integrated as part of the Q-Flow library we have however like for other custom tests we actually make a plugable so you actually could go to the type of communities if you build any new custom test you could just plug in just like how you build other Q-Flow problem components you just find that custom test what Q-Flow problem component you understand and you can just run it on top of Q-Flow problem with tactons that's how we actually working right now thank you very much is there other question if not thanks everyone thanks for being here thank you thank you very much