 Hello, this is Naohiro from Fujitsu. I'm going to talk about deploying OpenHPC cross-platform cluster by using the tool we developed. So let's get started. My presentation consists of three parts. First part is introduction and background about OpenHPC R64 server and the tool we developed. Second part explains the problem of single-platform cluster when we try to introduce R64 server to existing x86 cluster. And the solution we propose cross-platform cluster with Naoh. We may call single-platform cluster homogeneous cluster and also called cross-platform cluster heterogeneous cluster. But we only deal with two CPUs, x86 and R64. So I use the word single and cross. The third part is closing and recap. Target audience is those who are interested in deploying R64 computer node in OpenHPC. Scheme level is entry. And I'm going to try to use abbreviation as a regular as possible. If I use it, I try to make comment on it. The goal is that I think these can deploy OpenHPC cross-platform cluster using QANU. That is SMS, system management server, x86 manages both computer node, x86 and R64. I listed all of the commands in the test file and I project it into my github. The following URL shows you all of the exact command without any omission for your representative. So you can recap it anytime. I summarized OpenHPC in one slide. In my understanding, OpenHPC is collection of HPC open source software package. So as not to conflict each other, there are many, many HPC libraries and tools. Some of the components can be confined in different ways by combining compiler and MPI runtime. MPI runtime can be either OpenMPI-4, Intel MPI, and Vapix-2 or MPI-CH. Compiler can be either Unionine, Intel compiler, or ARM compiler. For example, boot C++ Scheme library here. The first block and the second block are x86 packages and the third block and the fourth block are R64 package and the first block is compiled by Unionine with those for MPI runtime. The second block compiled by Intel compiler with those for MPI runtime package. The third block compiled by Unionine with those two MPI package. The fourth block is compiled by ARM compiler with two MPI runtime. And the resource management can be chosen one of three, SRAM or OpenPBS or PMI-X. Provisioning management can be chosen either while wolf or xcap. And OS is supported either St. OS 8.2 or OpenSystems-Leaf 15.2. In this talk, I took a combination of St. OS 8.2 and while wolf and SRAM and I use Unionine and OpenMPI-4 for demo. The current status in terms of R64 server, bad news is that R64 server is still expensive and not yet popular as opposed to this smartphone. And good news is that R64 HPC won top 500 the other three titles for two consecutive times, June and November this year. This fact proves that R64 is capable for HPC in terms of performance and power consumption. In terms of assembly, RAID 8.1 R64 is running on this system, Fugaku powered by A64 FX SoC. So R64 server is standardized by ARM server-based system architecture, SBSA specification. OpenHPC weekly resource page introduces cross-platform provisioning assembly for while wolf, such that allows creation and assembly of R64 images on existing X86 host using containers. The hyperlink point to the repository close SMS R64 shell we maintain. Basically, what is this tool for? This is a tool that is implemented as a Docker container to deploy OpenHPC cross-platform cluster by enabling to execute R64 binary on X86 machine. So in this session, I call it SMS R64.shell for short. OpenHPC install guide written for each CPU architecture. So if we follow the guide exactly, typical OpenHPC single platform cluster is deployed such as this page. In this case, admin must manage two clusters. That means two database, two user accounts, two set of second files and et cetera, et cetera. SMS R64 kind of redundant. Fourth, if we virtualize SMS R64 into SMS X86 physical server, certainly one physical server decrease, but admin still must manage two clusters, no unification. So this is not the cluster we want. If SMS X86 manages both computer node X86 and computer node H64, everything is integrated into SMS X86. So no deprecated administration. So unified database, unified user account, unified set of second files. So this is ideal case. This is a cross-platform cluster we want. So how do we do that? The answer is relatively simple. Just two steps. First step is set up SMS X86. Then second step set up SMS R64 environment on X86 machine. In step two, we make use of R64.chain. That is cross-platform provisioning assembly for one world. Open HPC web page provide install guide for combination of each CPU architecture and OS and provisioning software and resource management software. Here is a copy of open HPC web page. The install guide listed by combination of the provisioning software and resource management software. So we use some version of CentOS 8.2 and Warwolf and Slime combination. In step one, set up SMS X86 fast by following the install guide X86, CentOS 8.2, install guide with Warwolf and Slime. No adjustment is necessary to the steps described in the install guide. This picture shows just after step one has been done. In the step two, we use SMS R64.chain container and set up SMS R64 environment on SMS X86. So we need some adjustment to the steps described in R64 version of the install guide. This picture shows just after step two has been done. Yellow box shows SMS R64.chain container is created in SMS X86. Step one has seven tasks. I go through all of them one by one. The first task is user account distribution setup by Warwolf. This picture shows just after install open HPC, basic component on SMS X86. MySqlRD, chpd, tftpd, htqpd, Slime, ctrld, mngd, and nfsd are running and cdshare, SSH are ready to use. Cluster user account is managed as OSUser and distributed via Warwolf WWSH file command which distribute ETC password, group, and shadow. The second step of step one is user's home directory distribution setup by nfs. Create cluster user home on local disk on SMS X86 in this case and export it to computer node X86 by nfs. The third task of step one is BOS, base operating system creation by Warwolf. Base operating system is minimum set of sent OS 8.2 and the source of bootstrap and VFNS, virtual node file system. Base operating system X86 is created by WWMKChangeRoot command with BOS X86 root pass and openHPC packages are added by VM command with BOS X86 root pass. All packages are pulled from repository X86 in internet. The fourth task of step one is bootstrap image, assemble, and distribute setstrap, Warwolf. Bootstrap X86 contains PXE, kernel, and init.fs and it is created by WW bootstrap command with BOS X86 root pass. The fifth task of step one is VMFS, virtual node file system image, assemble, and distribution by Warwolf. VMFS X86 contains final OS run boot image and it is created by WWVFNS command with BOS X86 root pass. The sixth task of step one is development tool installation and distribution setup by NFS. Development tool X86 are collections of library and tools and they are installed by VM command into sms x86 slash OPT slash OHPC slash PAD and exported to CN x86 as slash OPT slash OHPC slash PAD by NFS. The last task of step one is computer node x86 boot. sms x86 issues IPMI boot command to computer node x86. Then computer node x86 initiates IPXE boot. Step two is almost identical with step one. The first and the second tasks are common with step one. It needs these two tasks if users are the same between x86 and R64. So let's go through the rest of the five tasks one by one. The third task of step two is BOS boot operating system creation by Warwolf using sms R62.check container. Notice that we installed docker into sms x86. This operating system arch64 needs to be created in sms arch64 container that is yellow box by WWMK changing command with BOS arch64 rootpass and OPT HPC packages are arched by young command with BOS arch64 rootpass. All packages are pulled from repository arch64 in the Internet. The fourth task of step two is bootstrap image assembly and distribution setup by Warwolf. Bootstrap arch64 contains PXE kernel and Enicolame OS. And it is created by www bootstrap command with BOS arch64 rootpass on sms x86, but last arch64 sms arch64.check container. The fifth task of step two is VFNS virtual node file system image assembly and distribution by WFNS arch64 contains final OS RAM boot image. And it is created by www bootstrap command with BOS arch64 rootpass on sms x86, but last in sms arch64.check container. The sixth task of step two is development tool installation and distribution setup by NFS using sms arch64 container. Notice that we need to use sms arch64.check again in batch shell mode, but interact in shell mode this time. Development tool arch64 are collection of libraries and tools. And they are installed by young command into sms x86 slash OPT or HPC-arch64 slash OPT or HPC by presenting slash OPT or HPC-arch64 so as not to override development tool x86. The path is exposed to computer node arch64 as slash OPT or HPC slash part by NFS. The last task of step two is computer node arch64 boot. sms x86 issue IPMI boot command to computer node arch64 then computer node arch64 initiate IPX boot. Now let's dig into technical details of step two setup sms arch64 environment task. In terms of sms arch64 environment I will show you how to execute arch64 binary on x86 manually, automatically and result changes. Then how to share container host file system with this. In terms of computer node arch64 boot how to make VOS base operating system and bootstrap and vfms virtual node file system then how to boot computer node x86 and arch64 using QMU. Finally I will show you how to manage job for x86 and arch64. How to execute arch64 binary on x86 manually? The answer is QMU user static. Arch command shows we are on x86 platform right now and pw command shows we are in base operating system root directory and file command shows vls is dynamically linked arch64 binary. In order to dynamically link arch64 ls command we need to invoke change root with QMU user static like this. QMU user static is user space interpreter for arch64 binary. The static binary is convenient because it is portable we can copy it anywhere without vls and vls path. Next, how can we execute arch64 binary automatically? The answer is dinformat-misk. The dinformat-misk is kernel function and kind of shebang extension. Defining magic number of arch64 binary in dinformat-misk coffee file in restart system b then QMU arch64 static binary automatically invocation is enabled. And we can learn arch64 binary on x86 as if it were x86 binary. Next, how can we execute arch64 binary without change root? The answer is dockercontainer. The arch64 QM arch64 static is added into container by defining docker file like this. Container host and guest are sharing the same kernel so binformat-misk setting is effective in container guest too. And sms arch64 docker container can be used as either interactive shell or virtual. And typing just sms arch64.shell returns interaction shell prompt so notice prompt is changed. typing sms arch64 with arch64 binary runs as batch shell mode. Next, how can we share container host file system with this? The answer is docker-b sms arch64 invokes docker with minus V option like this. Then arch64 base operating system image can be accessed from both host and guest through shared volume. Next, how can we create arch64 BOS base operating system? Here 3.6.1 is the session number of arch64 openHPC install guide. The answer is to type wwmk-changing-command in sms arch64 docker-shell container. Here first type sms arch64.shell then notice that prompt is changed. And then set BOS root path to the shared volume and copy qm arch64 static to BOS. And then execute wwmk-changing which create arch64 BOS in the shared volume. Next, how can we assemble arch64 bootstrap image? The answer is to type ww bootstrap command to sms x86 shell prompt. Notice that we are lucky in sms arch64.shell but in sms container host. ww bootstrap with BOS path assemblies arch64 bootstrap image but the attribute of the image become x86. So currently ww bootstrap command doesn't take minus a arch6 shell parameter. So we need to translate to arch64 by ww shell bootstrap set command. Next, how can we assemble vmfs image virtual node file system arch64? The answer is to type ww vmfs command to sms x86 container host. Notice that we are lucky in sms arch64 bootstrap in sms container host. ww vmfs with BOS path assemblies arch64 vmfs image but attribute of image become x86. Currently ww vmfs also doesn't take minus a arch6 shell parameter. So we need to update to arch64 by ww shell vmfs set command. Next, how can we boot computer node using QM? QM parameter is very complicated and difficult. So this is just for your reference. In physical machine environment we issue IPMI boot command from sms to boot computer node. The pflash is available on UEFI firmware. So see the slide later. Point us to our client resource page. The last, how can we manage job for each platform? The answer is to define partition for x86 and arch64 in slum.conf file. S inf command shows all partition. SCAR denotes default partition. We can specify partition to S1 command with minus p parameter like this. If load minus p parameter default parameter is chosen in physically. Now I'm going to show you sms arch64.shell demo. As I explained, sms arch64.shell has two use cases. The first case is to create BOS base operating system arch64 in interactive shell mode. Second case is to install development tool arch64 in batch shell mode. BOS arch64 creation. Container host sms is x86. Sealed volume is empty on sms and create a file in shared volume and check a file is created. Check sms arch64 in interactive shell mode. Notice that prompt has been changed and CPU arch texture become arch64 and see if the file is there in shared volume just recreated. Check shared truth to BOS root path, create user B in BOS copy QM user static to BOS user gene to execute arch64 binary start mk change root with BOS root path then start installation 191 package has been installed with no error. Check BOS root path, image has been installed check bls is arch64 binary exit container set arch64 ch root to the BOS root path and BOS image can be seen from sms and check ls command is arch64 binary development tool installation we are going to install arch64 the new line compiler sms host is arch64 x86 and arch64 the new line compiler is not installed on sms start sms arch64 in patch mode specifying the yam command as an argument installation started and now the compiler is installed on sms x86 and see the gcc binary is arch64 now I am going to demonstrate cross platform cluster here is the demo environment there are 4 CPU, 32GB memory and 2 ethernet one to internet, the other is to cluster network N1 and N2 are computer node x86 both are running as QMU virtual server which has 4 CPU and 8GB so computer node x86 has total 8 CPUs and one and two are computer node arch64 both are running as QMU virtual server which has 4 CPUs and 8GB memory so computer node arch64 also has total 8 CPUs sms and cn computer node sms is x86 and wwsh node list shows all of computer node n1, c2, n1, n2 and se4 shows all of partition there are 2 partitions x86, 64 has computer node n1, n2, arch64 partition has c1, c2 and I switch the terminal to n1 and n2 and the run boot OS image almost take 3.9GB and both node marking development tool path and home directory and cpu is x86 and number of cpu is 4 I switch the terminal to c1 and c2 and run boot OS occupied 4GB and the development tool is mounted and home also mounted and cpu arch64 is arch64 and the number of cpu is 4 and let's start MPI job on x86 create x86 directly so it's not going to mix arch64 binary. The wwsh node list shows loaded component currently the new line and open MPI 4 is loaded and MPI CC is located in the community directory that is NFS exported and see the hardware program and initialize MPI and communication channel and load and synchronization point and load 0 print hardware and the list of then print processor information and compile the policy and check the binary age of arch is x86 and the final job in interactive mode specify number of processor number of load and partition x86 now from the change to n1 and start the final job and all eight processors return the answer exit node 1 and see the MPI job definition file the number of load is 2 the number of processor is 8 then start prun command so put the batch job to the x86 partition and job number is 37 and job 37 status is running and now for completing the result is debugging the file and all eight processors return the answer now MPI job arch x64 create arch64 directory and load in c1 to compile the hardware as arch64 binary logic module and MPI cc is located in NFS tool folder and compile hardware c and 8.2 arch is created and check the binary is arch64 and execute parallel job as interaction specify the number of processor 8 of node 2 and partition is arch64 and now program will be c1 node and start prun and all eight processors return the answer exit c1 and start batch job specifying the partition arch64 job number is 39 status is running and still running and now completed the result is stored in the file then all eight processors return the answer now put the end of arch into the default partition so job is 40 and job is failed as expected because x86 machine computer node cannot arch64 binary so that's the reason the error is exact format error this page shows pointers to online resources for your reference recap my presentation so please remember two things I will show you how to do this the first is to deploy all the command line steps to replace using qm is available on github the url is here I hope you try it by yourself and if you have a problem please report the issue to github or send me an email thank you for your attention