 Okay, so let's get started. Hello everyone. I'm very happy to share the topic here. My topic is Open Standards for Machine Learning and Deployment. First, I'd like to introduce myself. I come from IBM. I'm Liu Honggang. I have been working in open source related items like open stack and also machine learning. Now I focus on AI solutions. I have two colleagues who cannot attend today's meeting because of visa issues. So both of these two colleagues come from Center for Open Source Data and AI Technologies. So Codate aims to make AI solutions to enterprises and companies. And this involves machine learning and also the machine learning workflows as well as the detailed treatment like data processing and also algorithm processing as well as optimization and customized services. So today we'll talk about one step of this whole workflow that is model deployment. What are the challenges and how could we solve it? We have three parts today. I will introduce the background first and then I will introduce the open standards for machine learning deployment and then we'll have the summary. First let's talk about what's your first impression when we talk about machine learning. Maybe it will remind you of some bad robots who want to slaughter human beings or those very good guys, good robots who will do good things to human society. I think machine learning is like a double sword. We learn from the data to make predictions and then you generalize the rules to be specific we learn from the historical data to make predictions about the future or learn from historical data to make predictions about the future in order to make decisions for the current machine learning for example for AI I don't think it involves a lot of human elements sometimes you can see in the movies some machines are friendly to human beings, they're not friendly to human beings AI is more like an intelligent system it's more like data processing and data match it's just a complicated match and correlation by using the current data to conclude the result and then give the feedback and then you further adjust your model this can be widely applied in smart city, smart farm, etc then I like to talk about the whole workflow of machine learning you may think we have a lot of data or the client's data and then we build a model we make predictions and then we can sit there and wait for the results we just wait for the results to come out but actually the actual process is much more complicated than what you have thought it involves not only a person or a team it involves multiple teams of different areas it involves their coordination and cooperation to complete this whole workflow first you have quite a lot data especially in the information age how to collect all those data ingest the parts that you think is relevant to you and I think for this part we use big data tools like Spark, etc and then after you get the data you take this data to the lab you write different kinds of models, you make predictions I think that's your direct impression of machine learning but it involves multiple steps and procedures how to visualize the data how to take out the features and transform it's not easy and then we train the model the model might have different architecture or structure and then you need to choose and you might have multiple versions of the model and then you need to control the traffic to the A-B test, etc and then after the model goes live you need to think about how well it works it involves multiple teams, not only one team and it's quite complicated this graph is very famous coming from Google it reflects the true machine learning situation, you can see the black area that is machine learning code and then actually you spend most of your time on other modules like to maintain or to pre-process after-process real-time monitor the system and last we'll talk about how to deploy the model completely that's maybe the final step but it's one step, sometimes you will neglect when you're in the lab when we deploy a model we need to think about three questions first is what are you deploying whether you use test flow or other means and then you need to think about where are you deploying whether it's in cloud, browser, edge and also you need to think about whether it should be real-time or streaming and the last question is how are you deploying whether you buy a server engine and how the model is presented what is a model? it's like a document you have the document and then you can predict actually it's not that way normally a model is to receive the data or sometimes when you upload a picture actually the model only receives some data because the model needs to do some pre-processing and then it will output the data and then the data will be transformed into a visible and identifiable thing for example a picture for the user so actually we have the pipelines not models you need to know what is involved for each pipeline but we have some challenges for example different departments may use different languages like data scientists or researchers they care more about what kind of problems will be solved by this model they care more about the what as for the performance of the model actually they don't really care but what comes to for example production department they will care more about performance for example they might need C language etc and also departments might use different frameworks how to make sure they can cooperate with each other seamlessly that's a challenge as well and also when you deliver the model for IT department they hope you can have as less challenges as possible and also the changes could be under control but for researchers or data scientists they need to make sure that their model is the latest one but for business department they don't really care whether it's based on container or other things they care more about whether this will create any profits bring stable performance to the clients so we need to reduce the differences or differentiation as much as possible and to make sure different departments can work seamlessly without these gaps and that's a challenge so for example I like to use test flow and others like use touch because I use pass and flow other use PR touch and then you have different customized ways then in the future you need to have a system that can support both of those different tools or languages and that will increase the complexity so for container that can solve these problems and container can give us packaging solutions and also make deployments they're easy to pick bigger and also they're easy to repeat and also different departments like to use their own methods and they are compatible but what about performance, the final performance of this container for example a model written by python can be huge but the performance may not as good as another language also API and also the formats it involves some issue of standardization for example different departments use different models but if department B need to learn from department A department A and if they can transform from each other then that will be easier so we need this standard so why do we need a standard when all these models are transformed into a standard format like when we were small we learned math now it's universal language for all over the world but for the Chinese traditional mathematics which are exclusive to our own culture it might not be applied to other countries because a unified standard can bridge the gaps between different formats and different models and give all languages with this unified standard we can use the same optimization deck we can use a single stack we can use the same tooling which makes it easier so now here comes our first open standard PMML based on SNL it was first developed and released in 1997 by a group IDM was one of the members PMML came very early and now there are more than 30 vendors and organizations using it mainly focusing on machine learning enduring but the spark process does not support this format but other exporters can do it and also a scaler an ARC an XG boost and light GBM also support their models transforming into PMML here is a format overview there are a lot of metadata in it names and formats it is friendly to our users applied to various scenarios now it supports the following algorithms but it has a problem one shortcoming that is too standardized that along with time we might need other things beyond the standardized aspect and which cannot be met with PMML such as plugins for example if a plugin was written by Scala it cannot be transformed into this PMML format because it does not support this new language also there is a question of licensing so for companies who want to commercialize it they might have concerns that they need to pay attention now the second is PFA PFA was was developed to PMML to solve the shortcomings of PMML it was also developed by DMG and it supports many customized functions PFA was written by Jackson it is different from PMML the input and output PFA is different it supports deductions and additions and also mathematical methods and also machine learning algorithms you can transform a PFI into a Scala class algorithm small functions and since it was written by Jackson so the cross language functions work very well it is a very easily usable user case the input and output are both very easy and also PFA supports two different status such as cells and force a cell is named value acting as a global variable used in an action and force it is closer in the concept to a database and it can be shared across action executions PFA the function database is very abundant including mathematical methods and also machine learning algorithms and also it can do self-inspection and static inspection and most importantly it supports our mostly used languages which PMML does not support but PFA also has it's a newcomer thank you it has some limitations it was projected by open data group so it covers scoring for PFA in JVM and Python at R well the advantages as to what PFA can do well are the type systems and also flexibility and also machine learning but it does not support deep learning well it does not support for generic tensors and also no awareness of GPU the tests are limited so these are this is a tool for IVM internal open source development and also a tool for input and output to leverage FAA and SparkML now that Arc is open source and we welcome everyone who is interested to contribute to it also developed by Scala and in the future we'll do some testing which to test if it gets of course deep learning now the last one is the open neutral network exchange supported by Facebook and other big platforms it's for computing graph and now it also has a subsidiary project focus on deep learning and tensor operations so with the release of the Onyx model so many other platforms need to give us more support and the Onyx input and output and rat time we're also supported by Tencent and here are also other supported tools maybe you want to focus on TOM TOM is machine learning stock which can give further processing and now it only supports static processing and cannot be used for further training which we must improve in the future to summarize it's very matril standard but it has good support for standard machine learning models and future processing it's relatively new compared with PMML it has highly customized and it can do pre-process and post-process Onyx is relatively new but it could be very good deep learning standard in the future so in summary open standards for serialization and deployment I think that's necessary for the current market but of course different standards all have their pros and cons it cannot, for example, a single open standard cannot cover all the items for machine learning there is no open standard which can cover all the problems for items so now we're thinking about how to apply these different standards into different scenarios or to use them as a portfolio to solve the clients issues so that's all of my presentation