 Thank you. Good morning. My name is Aaron. I'm the founder and CEO of embedded technology. Today I'm going to show our experience over the set performance in an ARM-based micro server cluster. Okay. Okay. First, I will introduce a little bit about embedded because it was a startup company. And then I will point out the issues of traditional server that running safe with multiple OSD in a single server. Okay. And how will we solve this issue by micro server based on ARM. And also I will show some the basic high-up ability and scale-out performance and some use case over the Hadoop. And finally I will let you know we use the ARM-based micro server. It will set up a lot of energy power for our server and how much we can set for you. Embedded, we are founded in 2013. We start with designing the ARM server for distributed storage. Until now, until now we have about five beta or more of storage now installed in U.S. and Europe. Yeah. And in 2016, our products are awarded as the best of interop storage winner. Okay. So first of all, why we are talking about ARM-based server for safe? The first issue is that we are quite used to the XAT server. That one server is very, very powerful and it servers many disks. For example, 8 or 12 or even more OSDs in one chassis. And the issue, the first issue is when the server is failed. No matter it is a meriting fail or some components fail, you will, you will, you will lost all of the data immediately inside that chassis. Yeah. That is, that is a big issue because you know, safe will have to its self-healing when it detects OSD dance. So this will make a huge network storm when it starts to recover or it will take very, very long time to recover. So this is a big issue that for not only safe, but all of the distributed storage. And safe is a network-based storage. It is not, the bottleneck is always not the CPU utility. We have customer experience that tells us, no matter you have 10 gigabits of Ethernet or 20 gigabits of Ethernet, you still does not have enough bandwidth. And the CPU utility are always very low. So CPU is very idle and not fully utilized. That's why there's also a lot of energy at waste. So the third is certainly the power consumption. Yeah. Normally 8 disks or 12 disks server as an OSD note, it will consume like 350 watts. Yeah. You pay the energy and for the power consumption. You also pay the energy for the cooling. So you pay double of your expense over energy. So the power consumption is consuming your budget. Okay. So how we solve this kind of problem by using a decentralized architecture of server, in state of a traditional one to many servers. First, in this diagram I would like to explain what is the microserver architecture. Okay. In our server for the SAF, there are eight independent ARM based server. Each ARM based server will play the rule of an OSD or it can be a monitor or can be a radar scale way. So to give the best bandwidth for each note or each OSD, from each microserver we have certainly a data storage for the data. That is 3.5 SATA hard drive or you can use 2.5 SSD for the OSD data storage. Also it has an independent SSD for the journal disk. So the journal disk is dedicated for that OSD and it is not shared with others. And on the server we also have third storage which is for the operating system and the SAF software is root 5 system. Okay. So the resource for any OSD is all independent, not shared with others. Okay. So they can provide dedicated storage interface, memory, CPU resource. And also the most important is the network. Okay. Every single server has a dual Ethernet. Each one is 2.5 gigabit per second. Okay. So how to make them as a cluster? We have built two in-chassis switch in a one-use chassis. The switch provides in cluster the in-chassis fabric for the OSD appearing and also provides uplink for the scale-out and client access. So to fully utilize the 8 server by 2.5 gigabits Ethernet, we have to provide 4 gigabits of uplink. So this is completely eliminate the bottleneck of a SAF cluster. Yeah. So this is also very important for a SAF performance. And in this architecture, these two switch are redundant for each other. Okay. So once the number one fails, you still can access from number two. So it is redundancy. And if any note fails in this chassis, you will lost only one OSD. Any other 7 servers are still working very well. Yeah. It never affects each other. This is a picture of the server. You can go to the right-hand booth to have a look at it. Okay. So as I mentioned, there are 8 microservers. These 8 microservers are all independent. And there are 1 M.2 SSD dedicated for that server for its OSD journal. Okay. And the one server has its own hard drive, not more than one. Okay. And all of the modules like here, no matter it's a hard drive or the server or the switch are all haswappable. Certainly include the power supply. Yeah. So it is truly no single point of failure server for SAF. And also a very important, because this is on server, is very, very low power consumption. Only 60 watts per unit. Yeah. Compared to similar X80 server used for SAF, you are considering at least 350 watts. Okay. So compared to this diagram show you the difference of one to many server, one server to many OSD or versus the microserver of one to one. When the traditional server, you have many disks in one server. So when you have one server fails, as I mentioned previously, all of the disks will be lost. Yeah. This is a very, very big problem. But for the microserver architecture, this is one to one architecture instead of one to many. So when you lose a single server, you just lost one OSD instead of so many. Okay. So the benefit of using one single node to single OSD architecture on SAF, the first benefit is that it is a decentralized architecture that distributes failure or minimizes the failure domain to a single OSD instead of single node. So that's a big node. It's a micro node. And the NTPF of an on server, I can show you the server here. I have carry one. Okay. Okay. This is the on server. It has very, very simple circuits, very, very few components. That means it has very, very high NTPF compared to a thousand components in a motherboard. The chance of failure is very low compared to an xxc6 server. Yeah. So with a dedicated hardware resource for the OSD, you got CPU memory, the storage, the network, and also important, you have dedicated journal disk SSD for that OSD. Nothing is shared. And the aggregate network can give you a very, very high bandwidth in a single one chassis, which is four 10 gigabit per seconds. And the low power consumption is another very big advantage. Not only the OSD can serve on this micro server, we also use the server for monitor and rados gateway and even isogasi gateway. So you need no extra hardware to make a cluster. And the basic high availability architecture is to have three chassis. That means you put one monitor in each chassis. And if you use replication, you can use crash map to distribute your data replication to different chassis and in different node server. Okay. So start from basic three chassis, which is 24 nodes. You will use three nodes as monitor and other 21 for the OSD. If you want to add more or scale it out, you just add more chassis. And after the first three chassis, certainly you don't need extra monitors. Then you can utilize all of the eight servers as your OSD. We did many performance tests. And today I will share some of the test performance with you. The first is that the setup of a cluster, a safe cluster test performance. We need a lot of client server to squeeze all of its performance to know how it performs. Yeah. For here, we have five Intel Xeon server for the client as the loader of the test. Through the 10 gigabit switch, and we use three units of micro server, un-based micro server with 21 SSD and SSD journal and three monitors. And totally, we create 40 RBD. These RBD are for all of the clients. One client has his own RBD and make the FIO test over it and aggregate the total bandwidth from all of the servers, client servers. Okay. So first of all, this diagram shows that the scale out performance. We start from one chassis, which is seven OSDs. And add another chassis to have total 14. Then add the third to have total 21 OSDs. You can see the IOPS, the IOPS of read and write is very, very linear scaled from, for example, this is 9,000. This is almost 18. This is almost 27Ks for the writes. Yeah. So can achieve this is that proof the network contributed a lot. Yeah. There's no bottleneck on the network. So we can do that. And also, we make a test over different uplink channels on the network to compare if we have all of the customers. Every chassis has only two 10 giga or utilize all of the four 10 giga network. What is the difference? Okay. So here we do some 4K random write test with increase his clients, number of clients. Then we compare its aggregate IOPS of 20 giga uplink and 40 giga uplink. It shows that average you get 50% of total bandwidth increased. This proves that the network always be a bottleneck of the safe cluster. So to have a very, very fulfilled network for OSD server, which is very important. Okay. And this is an experiment or test that we did to know how fast if an OSD fails that it can rehear to completely healthy. So we have, we use 16 10 terabyte hard drive and replica two pools. And we rise totally around 48 terabytes of data. So it's average. Each OSD has three terabytes of data inside. Then we suddenly turn off one OSD to see how it will rehear. Totally, we used five hours and 10 minutes to have that disk. The data loss in that disk reheared to all of the other 15 OSDs. So compared to traditional disk array, it will take a very, very long time, more than 41 hours. It is two and a half days to just recover one disk. So safe is very, very powerful because all of the OSD will take over the reheal. If you have more OSD, the time of recover will getting less and less linearly. We can show you that we can, it scale out linearly. That means when you have disk fail, it will reheal multiple times. You have multiple OSDs. Okay. Next, so I would like to show a use case in a telecom company in Taiwan. That is the biggest telecom company in Taiwan. They have a cluster which is used Hadoop file system for the big data analysis. The scenario is that they collect different data source from everywhere and connect to a staging server and convert it to three copies of HDFS. And each HDFS has its data node. Every data node is a server and there is a computing node with storage. So it's like a hyperconverged infrastructure. Computing and storage are combined together. But when they grow their cluster and the storage gets bigger and bigger, up to now they have more than two petabytes of data stored in that. But they still grow very fast. The issue is that the HDFS, when you want to scale out your storage capacity, you have to scale out your computing as well. That is very, very expensive. The computing power, they're ready very enough because they have hundreds of servers. So they want to separate the storage and the computing node. So they need an external storage to store the data. They are not expect to have a very high performance external storage. But they can tear in their data to have hard data still use HDFS. But the history and the code data were stored in some other ways. So we tried to work with them with many different ways from SAF to native works is Hadoop. If you find the papers and you get internet to search, you will find several ways. First is the Java protein for Hadoop. And it works, but the performance is very poor because SAF is based on C and Hadoop is based on Java. So the performance is very poor. And we also tried the way of Amazon like S3A. We make it work with S3A, but its performance is not good as well. So finally, we use SAF file system as its local file system. And then it is not 100% fulfilled their needs, but it solves a lot of their problems. Yeah. To solve the computing storage can separate with the storage and reduce their cost. And when the server, because they have Hadoop file system, the disk is inside every computing node. When any disk fails, it's making a lot of trouble. You have to immediately to replace the hard drive in the server. So that's a very high load of operation and maintenance. So we work out a way to use the SAF file system for the Hadoop. Before we make the production, we have to make a proof of concept in the lab. So besides the SAF file system, as mentioned, we have work on other solutions like S3A or the Java plugin. No others work except the SAF file system. We use it as a local file system. We make a small benchmark. So in the benchmark, we compare to Hadoop file system. One is use three or four virtual machines, but give it a physical link to the physical disk to simulate what's the Hadoop in its environment. Because they are using a bare metal for the Hadoop analysis. Compared to use the SAF cluster as a local file system, we use the test DFSL to measure its throughput on map reduce on a Hadoop computing node. The simulation is that we give three units of our micro server, which has also 21 OSDs in a small scale. It shows SAF has faster write performance in some way compared to the blue line of Hadoop file system. The test with various file size or number of files. Most of the test shows better performance. But for the read, we got a slower performance on SAF file system. But this is not an issue for them because they want to have a trade-off between very high-expensive and very high-power consumption HDFS server versus our low-power server. We got better performance over the write, but a little bit lower performance for the read. Our product is a combination of three key components. One is the ARM server and in the center, in the core, is SAF. Currently, we are using dual as our SAF. And the third, on top of SAF, we have a unified virtual storage manager, which is a web user interface for the user. We make this because when we promote our product to our customer, many of the customers, they have an issue that they say, yes, SAF is very powerful, very good. I would like to use it. But I don't know how to use it. Are you going to ask me to learn command line? We made this problem. So we come back to think and decide to make a SAF user interface for all products that make it much easier and lower the barrier, the skill of the user. Yes. So make it very simple. Later, I can show you a video about one minute about our user interface on the video. And we are demoing in Red Hat booths. Welcome to come to the Red Hat booths. We can talk about if you have any questions. And our product was the winner of Interrupt 2016. Yes. So now we have our ARM server. What is the next? Because two years ago, when we designed this server, we have only 32-bit ARM server chip to use. And most of the software, including SAF, is not supported by, does not support the SAF. And so does not support ARM. So we have to do most of the things from zero until now, everything by ourselves. Yes. And now the 16-bit ARM has become reality. So we are ongoing to design a new platform with same form factor, but it is 16-bit ARM server. That can give us more memory to use. Because now we have only two gigabits of memory. We have to very carefully to use all of these limited memory, but it's still, even it gets very good performance, but we think we can make it much better if we have more memories. And also, for the SSD or SD, if we have powerful CPU, then we can make it almost the same as the Intel server, no difference. And because we are also using the server as its gateway, so we will have better performance on a gateway like Redo's gateway. And finally, certainly, we can use most of our distribution of Linux and SAF, including Red Hat or Sentos. Yeah. Okay. So finally, I will give you a demonstration over the WebUse interface. Okay. This is an interface of Login. From the first node to create the cluster, you can use this interface to create your first monitor. And then you can use it to create multiple OSD at a time. You choose if you have a journal or not. Our default is journal. Okay. This shows the MDS. You create your MDS. Okay. This shows the crash map. You have a visualized chassis and rack. And you moved your host to that chassis to define your crash map. Okay. This is the pool creation. You can create a replica or a erasure code pool. Also, you can create image. And we provide the snapshot and also tiering for pool. So everything is by WebUse interface. This is the authentication of SAF-X user. And then you can edit its compatibility. Then you can download your key. This is a SAF file system MDS. Okay. Create, assign your pool for your file system. Then you have active MDS and file system. This is creating the backend pool for OpenStack. S3 and Swift users. You can create as many users as you want. At the quota, this is the dashboard. And several other management functions include NTP and OD log. Thank you. Thank you for your attention.