 OK, good morning, everyone. So thanks for joining my session today. So I'm Kevin Zhao from Leonardo. And this presentation was being previously called Speaker by me and Xinliang. But due to some reason, he cannot be here. So I will take this presentation today. So I want to elaborate some HPC storage status on the ARM 64 environment, so especially on the ARM 64 servers. And first, yes, I want to do a simple introduction about Leonardo. So Leonardo is an open source organization who works with the business and open source community to develop the software on the ARM-based technology. So from us, we are a collaboration organizations by with ARM and its another some semiconductor vendors. Generally, the ARM and its customers. Because unlike the X86, the CPU was being manufactured by Intel or AMD themselves. So for ARM, ARM offered the IP model, and the different CPU vendors will offer CPUs. So there will be a very potential opportunity for the collaborations between the ARM and other ARM 64 CPU and semiconductor companies. We have some low-level software collaborations. So that we have founded this. We have actually, ARM and its partners has founded this company from 2010. And now we are actually in the last Linux upstream. The 6.2, we are the top one Linux contributions. So today, so my talk is generally very simple. And I want to give some brief update on our work on the HPC file system support on the ARM server side. So I will briefly introduce why we do this and what we have planned and also give some update for the other three projects for the Luster, Dels, and BGFS. So we can see that in the HPC world, there are generally two famous ranking lists. So the top 500 and the IO 500. So for top 500, we can see that it is really to see the ARM servers. Even though that the Fujitsu Fukaku has been ranking in the top one for maybe one or two years, but we can really to see more. And in the IO 500, the problem is even worse. So we can see that in the top 100, there is no ARM environment. So that's why. The problem we observe that is one big problem is the backend file system, the backend storage support on ARM has not been well supported. Because in the HPC area, it is a generally dedicated area. And it has been dominated by some open source project, but that one has not been well supported. So that way, why we address this problem? So we can see. So we choose why Luster, Dels, and BGFS. Because in the top 100, we observe that around half of the top 100 in the IO 500, they are using the open source solution, Luster, Dels, and BGFS as their backend file system. And to address, to fill this gap, so we have set up an open project. It is a collaboration project. We have collaboration with ARM and Hi-Silicon. So here I listed a project page. You can see more details, project information here. And in this project, we are doing the engineer work, the open source enablement, and the optimizations for a dedicated area. So the first one is HPC file system. I will elaborate today. And the other two is a traditional storage file system. So mostly, we are co-working with the CIF upstream to support the CIF ARM 64. And also the cloud native storage, yeah. So we have done some cloud native storage ones. For example, the mass store, and also the Rook to support the, to enrich the ARM ecosystem. So and also, if you have some more questions in the storage status on ARM, please feel free to post the information on the open discussion list. So yeah, so first one, Luster. Generally introduced Luster is a framework with the client and server. So the background for Luster is, so Luster client on ARM was being already supported several years ago. So that it means that you can use the Luster client to connect it with a remote Luster cluster, which can be offered by the X86 backend storage. And those one has already been adopted into the AWS service. So now the AWS has offered the Luster client to ARM 64 services now. But the server side, they support are just finished by Linaro and the Luster community. So I will give more discussions, give more upgrade here. So in the server side, it's generally running the metadata service, the management service, and the object store service. So the first is ARM 64 support and CI. So that one has been actually already finished by Linaro and Luster upstream. So if you want to try the ARM 64 Luster server and also the Luster client, we have already offered the ARM support. And you can go to the Luster download. And the main page will redirect it to the Linaro repo. And we will also set up the Luster ARM 64 builder and the CI. It is a daily running CI, which has been maintained but in our data center. And I will introduce more in the CI side. So the CI side, this is a very simple precise. So I just go through simply. So just grab on building every day and upload the RPMs to the right repo. And the other one is a follow-up for the builder CI. It's a testing CI. So for testing CI, the process you're already using the three-phrase. And we use the Terraform and OpenStack to provision the cluster every day, install configuration, and running the Luster testing. And then after that, so we will post the testing data to the model DB. The model DB is maintained by Luster upstream. So this is a panel for the different developers to see the CI status. So that would be very nice for the developers and users to see the current status and address different problem. And this is the test suite. So I just leave some information here for test suite because Luster has been already into production for maybe more than 15 years. And it has developed a lot of the test suite to cover different scenario. So the top four was a basic function test suite. And for the regressions, the multi-client and the concurrency, and the failure regal race, and also there are configuration and deployment test suite. This one, actually, before we have fixed a lot of the bugs for test suite to make sure that the Luster server can work well. And actually, this work has to still work in progress. And we still need to identify more issues in the test suite because there are usually more than 2,000 test suite for a simple testing run every day so that this work has to still work in progress. And so Luster, it has a style that previously it was developed a lot of the sanity-related testing for the beta function. And also, in the following years, Luster want to address the new storage class. For example, the SSD, the NVME, so that it has also support some new features. For example, I see here, at least here, is a hierarchical storage management, the failure redundancy, and also the persistent client cache. So that one also, for the dedicated new feature, Luster community also write a new class, a new suite, to cover these different features so that in this, we call it advanced features. We can also observe some 64 failures. But fortunately, we have fixed almost all of them, but still need to address more. So I live here about the remaining issue. And yeah, so this is a critical bug fix for a critical one here that is a crash or cluster hon. We can obviously very easily to see that the Knobs or the testing hon for the generally running in the Luster server on the M64. So we have sort of the priorities here. And also, the top three of the crash are generally related with maybe the data structure alignment, which is different with ARM64 and X86. So the other one is a 64K page related. Because this one, the 60K page size is, I don't know if it's clear, is that this is only introduced in the ARM architecture. And usually in the X86, the 4K page is quite popular. And previously, in the ARM ecosystem, the CentOS 8 has using the 64K page as a badie for it so that some of the software need to doing some of the upper layer software need to doing some cold change to adjust this. So that in Luster, it is a kernel mode file system. So it also has tried to support the 64K page size. And also, Luster introduced a page size translation between the 4K page server in the X86 and the 64K page client on the ARM client side. So it has actually introduced some new problem. I live here for the strapped directory and also the red stock. So that we have also fixed this. But from our point of view, we don't know to advise the user to use the 64K page in the future. Because currently now, most of the operation system in ARM architecture side, so the CentOS 9. So the 1, 2, and so in China, there is another one called OpenULA. So they have changed back to 4K page size now. And the last one is OpenZFS. The OpenZFS is back in the file system, which has been leveraged by Luster in the server side for the metadata service and also the OSS. We found that the OpenZFS checksum algorithm, this is a body-forted open features here, it'd be a hot point. And the performance is not very better for their previously neon support, so that we have doing all optimizations to increase the performance about 30%. Yeah, after the enablement, the new problem here is IL-500 test. So we have using the traditional configuration for 10 node, so the file server and the file client. So after we increase the high concurrency for Luster, we observe some new problem. It's still the critical one, which induces the kernel of OSS. So I left it here. Those one were actually dedicated with the ARM architecture, so if you have more information, you can come to this link to see more information. And the Luster 3 ones was related with the configurations. For the OpenZFS backend, we observe the OMQ. And also for the testing, we have also testing the environment for when you run an MDT test and then using Control C to destroy the cluster. And we observe, obviously, CPU software lockup. And after that, we have solved this problem by changing the RPC. And we found that the problem is RPC configurations. So we have changed and recommended the information here. And also another one is using MPIL to access the IL-500 test to try to achieve more regular testing and better performance. Also, so we have fixed some more testing suite issues. Much of them are related with the hard-coded or page size issues in the Luster testing suite, because previously, that one was dedicated and designed for the X86. So there are still some more testing suite issues for the recovery, the persistent client cache, and something else we are still working on. And the last part is open EULA support. So the reason why I listed here for open EULA is it is another operation systems, which is quite popular now in China. And now it's a 35% margins in the Chinese market this year. And for Luster upstream, previously, we offered packages for the CentOS-8 stream. And we want to have a more stable operation system. And we want to make the Luster to be available with so-called can support operation system seek. So that we have collaboration with the open EULA SDS seek and offer in the package and maintain the package. We will also pass the Luster testing according to the open EULA release schedule. So now we've finished the Luster client support in open EULA and continue to finish the server support on the EULA. By supporting this side, because of open EULA LTS-20.03 are offered the 5.10 kernel in its kernel, which is quite different with CentOS-8, so that we've also backported the Luster LDCFS patches on the LDCFS so that to support the 5.10 kernels. And also, this work was being finished by Linaro and the Luster upstream. Also, the dependencies, yeah. Because previously, we are testing the Luster on CentOS. So on CentOS side, they open the ZFS and also the E2 file system progs. So E2FS progs, those one was already being supported on the CentOS, but that's one has not been supported on open EULA. So even though open EULA is an RPM-based file system, sorry, even though the open EULA is an RPM-based operation system, so that we've done the build support for open ZFS and also the E2FS system progs in the open EULA to make sure that the Luster server and client support can be addressed in the open EULA. And this one will be released on January 13, if I remember right, in the open EULA 20.03 LTS-SP2. Here is the engineering work. So I don't want to go through very, very detail, but our target to move to open EULA is the Luster version 2.15.x stable release. And next one is the first top priority for us is switch the Luster CI to 4K. Because when we are work previously, it was based on CentOS 8 on running the daily test. We found more issues, which would be introduced more complex to support the Luster on open to support Luster on our 64 servers. And we'll talk with different community and upstream. And also the customer side, much of them does not have quite interest on 64K page so that we will move to 4K page to solve the problem. And the other one is continue to fill in the Luster on 64 test suite. So there are still less than 15 test issues we're trying to fix. And then DELS is a so-called distributed asynchronous object storage. So this one was previously supported by Intel. So Intel has promotes these solutions basically for their non-volatile memory, not only the NVMe, but also the SCM, the storage class memory. But so unfortunately, we know that Intel has dropped the bin list of opt-in. But the DELS upstream, currently now, is still there. And also it is a popular priorities for the Intel HPC war because the common server in the HPC market, Intel also want to propose the solutions for their server with NVMe and the future of their server with, for example, the UCI-based memory. DELS now previously was designed to have very good APIs and data structures with AI workflow and also the big data workflow. Here, I listed the DELS public community roadmap because some of our community members also have this problem is if the DELS will continue to live, even though the Intel has dropped the opt-in bin list. And we can see that from the DELS two down four, the upstream has plan to move their metadata management service from their persistent memory to the NVMEs first. Because Intel had dropped the opt-in, the storage class memory, but they still want to make sure that those projects are not be bounding with the opt-in memories. And those work will be finished maybe one year around the two DELS versions. And in the future, we can see that the upstream has a tension on the CCHI or UCI-based memories. Because this one was quite popular technologies in the future for the storage. So what have we have done for the DELS on ARM64? So first one is a build. So that was an Intel-based, Intel-controlled community. But they are open to support more architectures. So that we will cover with them to fix the data structure check issue, the SPDK build, and also a dependency for the IPM control. So this tools is, the IPM control is a persistent memory control libraries. So those are only running on Intel side. And other simple supported and other DOP packages and the telemetry. So after that, the DELS has supported build on ARM64. So since the DELS 2.3 version. And also we've set up the DELS CI. The DELS CI now has been included the two builders for IPM1, for Open Susy Leap, and the ARM Linux one. The DB builder is running for the Ubuntu 20.0, 22.04. And also we've set up the node local test CI to doing the landing testing after each patch merge. So in the red picture, it's a landing test build. We can see that it will be running after each of the PR merges. And if you have more informations, we have read a wiki in the DELS wiki side to record more informations. And yeah, so the IL-500. So those one will be even tricky for the DELS ARM64 because we don't have the persistent memory on ARM. And all we have is the first is the NVMN. The NVMN is the memory with a disk with a battery. So this one cannot be a very big capabilities. And also for the testing convenient, we're using the real memory to emulate the ICM. That mechanism has been officially supported to make sure that testing will be simple. So that has introduced a problem I listed in the last part. So the first one and the other one is a remaining issue is a 64K page size crash. So we have now fixed this. And after some debugging, we found that it will be a tricky issue and also somehow, and much of the customers now does not have the 64K page interest so that we will just live there. So yeah. But for a 4K page, it's supported well. And the IL-500, exception exists, is much about the capabilities of our emulation because the memory-emulated ICM's capability is very small. And for DELS, it has previously supported. And suited for the big-size storage class memory. And usually it will store the small block less than 4K and to the ICM by default. And also it will also, for each target, it will also allocate the memory, the 2G memory ICM and the 10G NVMe. But it will be very easy to consume all the ICM so that we can very easily observe the IL-500 exists. So after changing those two, we can get a validated result after changing this. So after changing this, we can get comparative performance on some unassisted full servers compared with X86. But now, at least, it's a BGFS. As a BGFS one, we've done something. The first is address the very critical issues on the AMV9.2 version, CPUs, and the copy from copy to user. So that one is bound to be related with two-arm features called PAN and UAO. So the time is limited. So I don't want to brief where a detailed introduction, but all the information has been recorded in the BGFS 7.3 release note. So now from the BGFS 7.3, the ARM supported has been officially announced. And we've covered with Heslican and SimpleQ to doing the performance evolutions to run the IL-500 test on ARM with NVMe and InfiniBand. And also, we can get some better performance with the parameters changing and the Neuma tunings. So all data information, we've published performance evolution by papers. So yeah, it would be very easy to gather more information if you have more interest. Yes. Yeah, so that is all my presentation for today. So any questions? Any questions? OK, thank you all for joining us today. Yeah, have a good day. Thank you.