 Hello everyone. I'm very glad to be able to attend this year's open source submit in North America The cloud submit it looks pretty cool Today, I will give you an overview of our open source project new us It's origins current progress and then the status of our community So here is my Topic today move us build up the unstructured data service So guys may be heard about the terminology RDS Which is the relational data service that is for Traditional structured data today. We're talking about the emerging and structured data service so first Let me introduce myself. I'm dream cool Zillis patterner technical evangelist. I'm also a voting member of the technical advisory committee at the RF AI Foundation Period to join Zillis. I worked for ICBC IBM Morgan Stanley and Huawei I had been a database engineer for 14 years in these two years recent two years I turn to act as a database product manager Okay, a little bit more information about our company's list So who is this? Zillis is the aberration of zillion of zillis So you could know by name that we are engaged in data related technology We are a technology startup company funded in Shanghai We focus on developing data science software based heterogeneous computing We drive our software business models through the open source Our vision is to reinvent data science To provide data related technologies for new domain new scenario new requirement and to Help people better discover the value containing data So our open source project mules has now joined the RF AI Foundation as an incubation project So Zillis is now a major contributor to mules project So talking about the data we generally Divided the data into three major categories. The first one is structured data Including numbers dates strings and so on The second is semi-structured data mainly includes tax information with a certain format Such as various system logs The third one is so-called unstructured data Like pictures video voice natural language So these are not easy to be understood by the computer So relational database traditional big data technology These technology are to solve the problem challenge of structured data and semi-structured data Can be handled by text-based search engine Only the unstructured data which accounts for around 80% of the total data has been lacking Effective analytic methods in the past So into the rise of AI the deep learning technology in recent years the unstructured data Analysis is accelerated So the charm of the deep learning model is that you can it can convert to the unstructured data Which originally the computer is difficult to handle into the feature information That the machine is easy to understand So usually the output is like the vectors matrix shown here So the analysis of unstructured data is transformed to Vector computation or vector similarity search this kind of thing So how do people usually use the AI technology to Analyze the unstructured data so here I give an example It's in it's a Flow-based AI application This is a very typical scenario. So assuming we have one video we want to Analyze we want to process so we can create some operation streams. So usually called pipeline the left most pipeline captured a video frames and then extracts the feature from the captured image So here for example, we can use the VGG model a model with excellent General generalization capacity capability. So at last we got image feature vectors in the left most pipeline So the middle pipeline handles sound so eventually it generates audio vectors that have converted from the sound and the rise most Pipeline automatically labels some attributes for the video. So if you have other special requirements You can also build a new pipeline to do related processing So this is why flow-based AI application are so popular because they are so flexible Developer don't even need to have to write code. I mean, there are web-based interface to help users to compose those new process Even if you don't have any idea about how to start this Pipeline you can always buy some Useful samples as a referral as I believe as a start But in this way, it also brings us a new challenge So the data becomes very fragmented So you see it was originally only one video But with the operation of the pipeline it was gradually transformed into different data Spreaded in different corners in different pipeline. So what shall we do? so let's turn to the Turn to the view to the traditional hierarchical view So we can see here both the top input and about in the But an output are Unstructured later the AI technology may lay functions at the middle two layers the green layer model inference and in the blue layer data service The task of model inference is to transform unstructured data into feature vectors models are trained But serving them efficiently is still not easy, but the good news is There are already some mature projects in the industry such as NVIDIA's tensor RT Intel's Open Vino Microsoft own onyx RT and recently Google is also Developing cancer flow runtime TFRT but There is no comprehensive solution for the data service layer So some people put those feature vectors in a structured database Others might put it in HDFS and then analyze those vectors through spark Also, you can use some in libraries So in this area, everyone make their own try attempt How to and that to analyze those are vectors? so the challenge is how to manage and analyze the vectors efficiently although a large number of trained Models are now available But AI technology is still difficult to go production because at the data service layer the cost is still very high So how to address this challenge? our answer is to build up the unstructured data service which is based on the Milvus project So it contains four parts the first part is vector similarity search which includes high dimension dance vectors, which is frequently used in deep learning scenarios and Also, there is sparse vectors in traditional machine learning scenario The second part is attributes. So those structured label Like streams so combine these attributes and the vectors we can provide the capability of high research So the third part is to support multi models as in the previous example a video has vectors of From of different dimensions. So there are image vectors and all the vectors So in the real world multimodal search is a general requirement So we need to introduce the concept of entity for the unstructured data So an entity could contain multiple vectors from different dimension or different models The first part here is the scoring component So in some scenarios like multimodal search We just mentioned because we introduced different models then the fully connected the layer of different model They might need to be further fused to form a new scoring mechanism for the final analysis of the unstructured data So at this moment Milvus already Has built up the built up the vector analysis capability. We are constantly improving and enhancing it We are currently developing the functionality of attribute filtering so we can support the hybrid search In this in soon, maybe it makes it the second half of this year So we are planning we we are planning the functions of multimodal and a scoring those two things are Relatively complex so it may come later. So eventually Milvus is not just positioned to be a high performance vector search engine We want to build a comprehensive infrastructure software for the unstructured data service based on Milvus So so maybe people you have been convinced that we needed a unstructured data service But you might still have this question. Why not to build it through? relational database or big data technology a Vector also looks like a number What's the difference between a vector and a number? So to be precise a vector is Consisted of a set of numbers the difference between vector and a number I think there are two major aspects first The common operation of vectors and the numbers are different. So numbers For example additions subtraction multiplication and division is the most common operation on numbers, but the vectors the most common operation is to Calculate the similarity So you see here. I'm giving the formula for Computing Euclidean distance You can see the commutation of vectors is much higher than the normal numeric calculation Secondly the index organization of the data is different So between two numbers The value can be compared with each other So we could create the number index based on algorithm like B tree But between two vectors we cannot perform comparison We can only calculate the similarity between them. So the vector index is usually based on algorithm like Approximate nearest to neighbor a an algorithm. So here I give two a an approach One is that the left side is the clustering index index based on clustering algorithm and the right side is graph index for example so because of these Significant differences the tradition database and big data technology are difficult to meet the requirements of vector analysis The algorithms they support the scenarios they target are all different So here is the big picture of Milvus I want to highlight four major parts here. So the first one is the Support of heterogeneous computing. So I mentioned earlier Zedis is a developing data-sized software based on heterogeneous computing So we have some experience in this area. So during Milvus design We thought about how to support different computing resource So that we could accelerate such a Computation-intensive scenario The heterogeneous computing resources supported in Milvus so far include For example SSE instruction set AVX 2 and AVX 5 1 2 Instruction sets on x86 platforms We also support NVIDIA GPU which is later than the PASCAM micro architecture The ARM 64-bit processor Now we are working with some partners About how to further extend Milvus to risk five processors, but this work is still in very early stage The second part is the Data management function. So we want to provide an unstructured data service So the function of data management is critical Milvus supports like data partition, data sharding deletion of vectors and also stream injection so you can dynamically insert new vectors into Milvus and it will create Index and dynamically for the new data once it triggered a threshold And then the third part is the adoption and improvement of the AN algorithm libraries So the capability of vector search is the fundamental function In unstructured data service. So Milvus can provide good vector search performance by Adopting and improving the well-known in algorithm libraries like face annoy the force is support for the Application development environment so to enable AI developers to build their applications on Milvus We provide a number of application development environment like Python most people use C++ because Milvus is Developed by C++ and also Java and in Java still a lot of business applications They need Java and the goal and like REST HTTP API So since new bus things vector search is a combination intensive scenario But it also has no for example like drawing Operation like in the RDS area. So it's relatively easy for distributed the deployment So here is an example of a distributed deployed Milvus We provide a proxy and then we can Distributed the data over different Milvus nodes So when searching we simultaneously Query the nodes or the nodes and the merge the result sets together before Returning it to users So although Milvus is a server So it's more complicated than always a libraries So but people are very curious about the performance comparison Since performance, I mean Impacts the hardware cost. So it really matters. So and they want to set an expectation So when they start to use Milvus, how what kind of performance what kind of hardware they need to Invest for this kind of scenario So we have run through the A&M benchmarks It is a set of well-known A&M benchmark tests So thanks for Martin Eric a lack for developing the benchmark test we focus it on GitHub and Run through all the test scenario on Milvus in several different public clouding environment like AWS Azure Ali-U. And we also ran the test on our local servers Including our notebook. I mean most people The data scientist or a developer they have a GPU powered notebook. So we we test a small scale Data sets on notebook. So, you know, if it's possible for the user I mean the developer to just use their own Laptop to do some application prototype and also we tested on a local server with two Intel X86 CPU So here I only captured one chart of Milvus But the benchmark test actually it generates a bunch of reports So if you want to find all those details You can visit our website Milvus.io. So just for the example here, this is the The QPS kind of QPS comparison between I mean most people asked about the if I just use raw face or I use the Face type IVF Index through Milvus. How is the performance difference? So We also want to evaluate how much System overhead in Milvus introduce in this scenario. So we we test this pretty seriously So you can see here if you are looking for a high Recall ratio then The system overhead can be it's very small. I mean You can see here those these dots Milvus and face they almost performed on the same level But if you are looking for Low recall scenario, I mean some scenario people may only need like 30% 40% Recall ratio then this time you will see The system I mean the server side overhead will will matters the will impact the throughput So for most people They're using the high recall ratio scenario like Larger than 80% 90% then the system overhead is relatively small So it won't be a big big challenge of when you migrate from face to Milvus All right, so this is how Milvus was born Our initial idea of this project was in October 2018 so that time we were involved in a project We need to deliver the vector search function We tried to do it in our Structure database or we have a structure database product, but it didn't fit fit well So that's why we start to think about this challenge seriously So on April 2019 we released the Milvus 0.1 and We tested it in our first city user and improve Milvus a lot So that time it was doing that an open source project because we are not sure if we are doing the right We have not to do it right or if it's Will be I mean if people will need it so We we tried our first to see the user and listen to their feedback So they give they give us a lot of confidence. So we decide to We decided Milvus 0.5 that release is a Pretty ready release for go public. So we So we put this release on to the github to to fully open source it so In this year March this year Milvus is joining the FAI Foundation as an incubation project so today if you check the Development activity webpage of Linux Foundation, you will see Milvus is the most active project from the software development perspective in the Linux Foundation. So Let me show you some Numbers about Milvus project so far Yeah Yeah, here is the current status of Milvus project We have made over 5,000 commits and 14 release So and we have currently hundreds of community users. So some of them have already put Milvus in production I Think that's Pretty I mean good thing. I mean Milvus is do a young project I mean less less than one year since it go over source But why people start to build their AI application upon Milvus and put it into production environment I Think the most attractive benefits are Two things first is it's easy to use and second it's fast which means You don't need a Very high cost hardware. It can be run with very low hardware cost So developer could make their Attempt with the pretty low cost. I mean those AI applications next page I will start to share with you guys the real-world use case You can think about why people want to adopt AI technology into their product Does AI technology provide the Call value or it creates some value add into the existing application or products So I mean those I mean the answer of those questions really Tells why they try to use Milvus to make these those try Okay So this is the first Use case It is an so-called intelligent writing assistant so the software vendor it's Their core business is to deliver these Office products That's top computer on mobile webpages So they want to increase the To increase the value add of their original office product and attract more use to attract more users So they design this intelligent writing assistant So it's supposed to help people to compose some kind of assing Like year-end work summary cover letter or refer letter so the software vendor first collect bunch of corpus data and after cleansing these data are encoded with taxi and then to Extract the paragraph and the cemetery summary Then they will be further encoded with infoscent model At last you get the vectors and they store them in Milvus So when any user submit the writing request It will go through infoscent encoder and you perform vector searching Milvus The search results will be further transferred to a draft assing So people can make some little Modification points and just quickly get a what we want Okay, the second example It is a Big data company. They have collected a lot of corporate credit data in China and also they collected around 55 million trademark images of Chinese company So they want to provide their members the function about search a company through the trademark image So they build the image search function upon fine to the VGG model and yours So since nobody know how many people will become their new members just for this new AI function search company Through trademark images. So the development and the hosting cost is very sensitive and Now they are very happy with the performance Milvus provided. It's around 20 minutes 20 milliseconds on the Cloud GPU server, I mean The Cloud GPU server at this moment is do not like NVIDIA a 100 those kind of doctorize the capability GPU the virtual GPUs at this moment is to Not perform as good as the physical one, but still they got the Performance result in need so So in these two Use case you can see the AI technology They're not the core value of the whole product, but they create some Value at point to attract more users to use their products but I mean at the beginning nobody knows if these function can attract how many people to join to become their new Users members. So they're very very sensitive about the investment on those functions How long it will take them to develop the function and What is the? Hardware hosting cost is all their concerns. So I think Milvus in these cases provide them a very low cost, I mean twice for for for try out their Idea about those air technologies and to see if those air technology will really attract more users to their for for their product if if those If those Functions didn't work as they expect and then they can just I mean put them away and Try another different AI AI technology in different functions so I Think that's why people These people use Milvus to create their applications to adopt more AI technology in their products because it's fast to fast to develop and Low cost to host So the last use case so So just mention like I just mentioned in the previous two examples the AI technology is not creating the core value but in this case Actually the efficient vector search is Creating the core value This is a pharmaceutical user They want to build up a platform to analyze over Billions of molecules to I know to analyze those chemistry structures so They first will translate the molecule expression into 1024 bits binary string data and a story to Milvus and Then they could perform like tiny model similarity analysis or Substructural analysis or super structural analysis. So Previously they are doing this with Spark cluster So to perform this kind of analysis They previously created a spark cluster to host around 30 million molecules So every search and I mean every structure similarity analysis will take around 14 seconds to complete but after we Support those tiny model similarity matrix in Milvus now with Milvus they can analyze over 800 million molecules within one second Actually 500 milliseconds on the single server with two way CPU So you can see the Vector search is Bring some totally different value into this scenario Okay, so yeah now it's come to the end of my presentation. So here is the useful links for Milvus project if we want to explore the possibility of introducing AI technology into your applications products Please think about Milvus It will be helpful. You can find our technical article our document and our source code in these useful links We always help welcome people to join Milvus community and Thank you guys for listening this Listening to this session. If you have any question, please let me know So it looks like we've got a question here. It's The GPU enabled version of Milvus requires NVIDIA Pascal GPis are there any support for OpenCL or AMD ROC to speed up the query? If not, is this on the pipeline? At this moment, yeah, Milvus only support NVIDIA GPU and the requires CUDA 10.1 Yeah, we currently Don't have any plan OpenCL or AMD ROC, but we will see if a lot of users need these kind of Support we will think about how to Implementing Milvus. Okay, it looks like we have another question here Do you have benchmark comparisons against other search engines such as elastic search? Yes in the Slides we have a pay we have one slide about the ANN benchmark That's available Milvus.io, but you know the ANN benchmark is particular for the ANN area the elastic search it's That's a different kind of solution. I mean that's most about the Tax-based search technology, so It's hard to compare between these two so we don't have a comparison with Elastic search regarding performance All right, another question. Can Milvus extract objects from unstructured text into a database in batch? Milvus is to Hosting the feature vectors the invading vectors the user need to run the inference model to extract those feature vectors and Insert those vectors into Milvus so that Milvus can help you to do the search. So I Think the question is asking about if like people user cam Insert the invading vectors into Milvus in a batch. I mean, yes, we support batch insert, but The feature vector extract is not performed within Milvus. It's through the model inference layer great That looks like all the questions so far we still have a few minutes left in the session So if you have any other questions put them in the Q&A chat Here's another No, never mind. It looks like that's all the questions that we've received so far. Do you have Any last comments you'd like to add otherwise we can continue the conversation on the Slack channel Okay. Yeah, just On the last page we have all those useful links so because We have limited time in this session so we cannot cover everything of Milvus So if you think if you are interested in Milvus projects so we can just visit Milvus.io website to get all the information you want Thank you guys