 OK, hello. Good morning, everyone. Welcome to join today's session. My name is Xu Quan from 99 Cloud. Today, this topic, I will talk about a collaboration we're working with Shanghai Jiao Tong University and Intel to build a bare metal Kubernetes cluster for HPC on OpenStack. And we are putting the translation medicine research workload in our bare metal Kubernetes cluster. So this is three parties' collaborations. Because of some visa issues, people from the daughter Luo from Shanghai Jiao Tong University cannot join this session to present with me. But I have another friend from Intel. He will join this session later on. He missed the time, so I will get started first. And he may join us maybe 10 minutes later on. And this is all the collaborators doing this project. Xuan Luo from Shanghai Jiao Tong University, Eli Qiao from Intel OTC, Jeff from Intel OTC, he will present with me. And Yufei and myself from 99 Cloud. So let's get started. Today's agenda, first of all, I will share with you the background of the Shanghai Jiao Tong University. I will share with you the cloud journey of Shanghai University, why they want to build a cloud, why they want to put the HPC workload on the cloud. And I will give you guys a cloud service overview of Shanghai Jiao Tong University and the reason they want to put the HPC workload in the cloud. And the last two parts, we'll share with you guys about the technical detail how to build a bare metal Kubernetes cluster in OpenStack by using Magnum and Ironic. And the next steps contain our thinking and some considerations during the whole solutions. First of all, cloud journey of Shanghai Jiao Tong University. As you know, Shanghai Jiao Tong University is the top five university in China. The university is majorly located in Shanghai. And he has many colleges in the university. A lot of compute resources, they are distributed because different colleges, they will purchase different hardware for their internal IT usage or their scientific research, which will cause the hardware cannot be centralized to management. And the hardware they purchase have very low utilization. For example, the CPU utilization will be below maybe 10%. And normally, they do not have a professional IT administrator to manage the IT infrastructure, such as the switches, servers, storage. So they will ask the students to help to operate the servers in their sparing time. So that will cause a problem is there is very high failure rate and energy consumptions when this college use the servers alone. Because the administrator or operator, they are not professional, so they cannot guarantee the SLA. And they will let each college, for example, the life science college, they cannot focus on their own science research, like the genetic research. They have to spend a lot of time working with the IT apartment to troubleshooting the infrastructure. So Shanghai Jiao Dong University, they plan to set up a new department to centralize, manage all the resources, which contain the physical servers, storage, and networking devices, and centralize them in a cloud environment to support the internal university IT requirements, and also, hoping those devices, those services can provide different colleges to meet their scientific research. So that is the problem of the Shanghai Jiao Dong University they meet. Also, you can see the data. We have so many colleges spending a lot of money according to the internal data of Jiao Dong University from 2013 to 2015, they purchased over 1,000 servers with over 40 million IMB invests in different colleges. So we can imagine if there is a department, they centralize all those equipment, they can reduce those costs with several professional IT department mentioned in cloud environment, and provide the cloud services to those college for their scientific research or for the IT hosting applications. So as I just mentioned, the mission of the Shanghai Jiao Dong University cloud services, they want to centralize all the hardware and improve the overall utilizations. And also, they can have some professional IT administrator to provide stable services for different colleges. And what's more, that is because the scientific research area have to require high-skill administrator, high-skill administrator to support them. Normal students cannot serve those responsibilities. So also, they require a very conflict-applic infrastructure to serve the scientific research areas. Before different colleges has different infrastructure, the information is located, actually is located in different silos by integrating them into a cloud environment. We can integrate the data from the whole university. So we can improve the collaborations between different colleges. For example, life science, they want to focus on how to research the genomic or some other life science problems. But they do not have high advanced knowledge on how to program or how to do data mining on different genomic. So they can collaborate with the computer science college to work with them together. But without cloud, they cannot do those activities together. By centralizing all the resources, we can have different colleges, work in different tenants. They can also have some shared resources together to share some research data. So we can improve all the scientific research results within the whole university. So as we can see, in a university, we have different workloads in a cloud. First of all, it's an education cloud, which means we can put a lot of teaching applications, university websites in those cloud days are very normal requirements, just like the normal ID hosting requirements. On the other hand, we have HPC workload. We have many high performance computing, such as the pie calculation or some benchmark calculation, genomic database mining, tax mining, such kind of HPC workload. Also, if we would like to have a quick result or response in the whole scientific research, the university itself cannot afford the infrastructures. Some HPC workload or some scientific workload, they may need to leverage some hybrid cloud capabilities. When we do small-scale calculations, we can do it in the university cloud. But some circumstances, we have to scale out to the public cloud, leverage the public cloud or some government providing the supercomputing environment to calculate all the scientific results together to return the result to the scientific professor. So as we can see, Shanghai Jiao Tong University cloud services actually have to provide these four computing services. HPC, normal ID hosting education cloud, can scale out to the public cloud, which means we have to enable the hybrid cloud capabilities. Also, we also have to leverage some supercomputing capabilities provided by the China government. But in this case, I will not cover the public cloud and the supercomputing. I will cover the HPC cluster and the normal ID hosting education cloud. So currently, Shanghai Jiao Tong University providing two cloud services. We call it JCloud and HPC cluster. Let's see what those two services. First, HPC cluster. This is a cluster that consists of 14 nodes, which install with the central S6.5. And since 2015, the user is increased by 89%. And this cluster provides services for the customer to use them to schedule the HPC workload. And the network can respond for high output. They use the infinite band over the old-fashioned driver to implement the high bandwidth network outage. The network bandwidth, I'm sorry. The total CPU and GPU utilization is over 60%. They also use the last with IMDA to serve as a fire system. The typical job running on this cluster is like a benchmark running link pack in the cluster and use the GTAC to calculate the dynamic. And some modular dynamic lamps, some CFD calculations using the open form. So that's the general status of the HPC cluster. The problem of the current HPC cluster is it is operated by a separate department. They focus on improve the utilization of the HPC and maintain the stability of the HPC cluster. This cluster currently is not managed by OpenStack right now. The second cloud service we call is a JCloud. This pilot seems to 2013 July with 15 nodes and full 10 gigabyte network setup in this environment. This is the first OpenStack education cloud in China universities. Currently, this cloud has already over 300 users and colleges to use this cloud. And normally, this cloud currently do not provide the high performance workload. It only provide the IT support and internal DevOps and teaching application hosting. So the next step, we would like to use this cloud, the JCloud, which is open stack deployment. We would like to use this cloud to manage the HPC cloud and provide and run some HPC workload in this OpenStack. So first, we pick up the research and we do a lot of selections on the HPC cloud. Which HPC workload? We would like to try to use the translation medicine research workload, putting this workload in this JCloud. What's the translation medicine research? I cannot tell detail of that because I'm not an expert of that areas. But as you can see, this workload normally is like a text search mining and like the database operations. For example, the genomic DB calculation and some disaster. It's not so networking intensive or IO intensive workload. We can put it on the CPU. Majorly, it's a CPU intensive workload. So we can put it in the JCloud to use OpenStack to set up a bare metal Kubernetes cluster to manage it. So the benefit of managing HPC in cloud, by using OpenStack, we can easily set up maintenance environment and provide self-services to the professor. And OpenStack is naturally supported. And we are more easily and flexible to manage the HPC cluster by just a click, send a command to the Magnum. They will help us to provide Kubernetes cluster and put our application in the cluster and running workload. And we can get the result from the volume. And by using the OpenStack cluster, we can centralize the HPC infrastructure and the cloud, normal cloud infrastructure and improve the overall utilization very quickly. So how can we adjust that? The solution is we have to pick up the HPC workload very carefully. But because some of the HPC workload, they are not naturally supported in the cloud environment. For example, I just mentioned some HPC workload. They are the network intensive or IO intensive that we have to set up those environment in the Cloud, OpenStack Cloud environment. That is hard for some administrator. And after we decide to put some HPC workload in the Cloud environment, then we can pick up an architecture or solution to figure out how OpenStack can manage HPC infrastructure. After we use OpenStack to quickly provision the HPC infrastructure, we can use the OpenStack to deliver the HPC infrastructure by using the pre-configured image or some templates by hit to deliver the HPC workload to the environment. So that is the background introduction. I will give you the technical detail implementation on building the bare metal Kubernetes cluster on OpenStack. So Jeff, do you want to share? OK. Excuse me, do you have a mic? OK, thanks. Wait. Hello, everyone. Just now, Shuchen has mentioned how the Cloud computing or the OpenStack can help HPC in the SPC area. Maybe the performance will be cared about a lot. So based on OpenStack technology, we need to build a bare metal cluster instead of a virtualization virtual machine cluster. So in the next several minutes, I will introduce some technical details about how we build bare metal cluster for the HPC scenario. So we can see we have several specific requirements for the HPC scenarios. So we can see sometime for the HPC use cases, we need the user program need to access some special hardware directly. So sometime the virtual machine cannot do that. So we must to run the program in bare metal directly. So that's one special requirement. Another one is the performance. HPC, the computing ability is a very central problem we need to care about. So as we all know, if we run the program inside the virtual machine, the overhead as our experience at least will be 7% to 8%. So it's better to avoid to use the virtual machine. And as we know, the containers can help a lot in the DevOps area. So for the maintenance convenience, it's better to put the program inside the container as well. So how we combine the bare metal and the container technologies? So this is our direction to think about the solution. And also for the deployment, if we want to use the open stack and container technologies, some maybe based on Docker, above the Docker, we need some orchestration layers management, our solution is Kubernetes. If how to deploy Kubernetes above the open stack based bare metal cluster, so our answer is two projects we need to focus on. One is the Meganam. Another one is Aronic. We can have a look about the OpenStack Meganam project. I think the Meganam project has been two years old. So I think everyone in the OpenStack community have heard about the Meganam. And maybe already know about the design details. So we can see we will provide Meganam. The Meganam services will provide API to the end user. And the API can forward the user request to the other OpenStack services to set up a tenant-based cluster to be ready to deploy the container orchestrations. The tenant's cluster is in the concept of the Meganam. And the user can deploy some container orchestration system inside the tenant cluster, the bay. The user can choose from the Kubernetes, the Apache methods, and the swarm. I think currently the Kubernetes parties are the most mature right. So we use Kubernetes inside the Meganam bay. From the diagram, another thing we need to care about is we know the Meganam will use the heat template to do the deployment management. And another project very important to our ITPC cluster solution is Ironic. Ironic window in OpenStack is also very, I think currently is now one of the core services already. So it's for the bi-metal provision management. It means the user wants to set up bi-metal from scratch. And after power on, how to install the operating system and the operating system, the image, how to build it, how to manage it, and how to choose it. So the Ironic will provide a total solution for the bi-metal provision. For our RGPC cluster, the first step we will use the Ironic's ability to provision all the RGPC-specific customized operating system images into the bi-metal. And then the next step to do is Meganam to deploy Kubernetes. Some of it can be considered a very natural solution for the SAP cluster, actually. But as during the deployment, we have found some very special question problems we need to address. Mainly, it has three points of the problems. One is the bi-metal networking model problem. Another is OS image is a choice. And then the storage problem, and we need also to be careful about. Mixing the first problem is the bi-metal networking. We know previously that Ironic can only provide the flat networking model for the bi-metal in the provision time. But I think from Newton, Ironic community has done very beautiful work to support the multi-pole tenancy network. So at the time point, in the time, we set up the RGPC cluster. Use Ironic, it helps us a lot for the multi-tenancy network. Yeah, in our solution, we need to use it. And another, about the multi-tenant network configuration, in our operation practice, we found that some tricky problem, it's a simple workaround. We need to go through the solution to let it work well. It's for the configurations. The first problem is for the public network, we need to set the external router to true in the configuration. And also, we need to set up the floating IP. Anyway, the float IP can also be disabled if we need not to access the external network. Another thing is the private network setup. The name, it needs to be named as a private, this word. Yeah, this is the book around. I think some bug is still in the community code. So next, we have also, after the setup solution, we have found a lot of bugs in the community code. And we can show in the last page, we have a bunch of patches to fix them. Another thing is about the bare metal operating system image choice. Currently, we are using Fedora operating system. And based on the vanilla Fedora operating system, we need to install several Kubernetes-related packages. It's also a lot of packages. I haven't list all the packages here. It's a long list. It's Kubernetes, Docker, and for some network setup components, a lot of them. So anyway, based on the Fedora and then when you install the packages, this is a customization practice. We need to use a disk image builder tools to do it. I think if the ironic developer knows it very well, so I just pointed out, these tools inside the, actually, this tool is from the ironic project. So it's the standard tool for the ironic user to do such things. And for all the different nodes, maybe some node is focused on the resource management, some other node to focus on the computing tasks. But for all the bare metal nodes, we are using unified images for common int. And for the bare metal storage problem, currently, ironic, I think I don't have no support for the sender remote values. So we can only use the local values. So we just mount the local disk device into the node. Here is two blocks to describe how to do such things. So if you have an interest, maybe we currently have no time to go through all the details. For all the detailed setups, step by step, they're very helpful. So I just list URL here. If you have an interest, I want to use it. You can check it out offline. What's the next plan for our RPC address, this setup? So I think this is our first practice for Shuchuan team, for SGTO team. But there's also a lot of things we need to improve in the future. First one is the node to get out the problem. Here, I mean the outer scaling problem. So currently, if we want to add more nodes into the cluster, we need to set up manually, step by step. Not very efficiency. So in the future, we need to add some auto scaling, scale out ability to let the ops work much convenient. Another thing is currently, we have no load balance support for our bare metal network. After the community work can be used stably, we can add the load balance support also as well. As I just mentioned in last page, the sender volume, the remote volume, we cannot use it. So it's not convenient in many times. So in the future, if Ironic can support sender remote value very well, then it can be help us again in the sender volume. And currently, for the operating system image, currently we're using Fedora. We know Fedora is normally for the developers. So for some really productive environment, the CentOS should be a better choice. We know in PSE, a lot of our company, a lot of our enterprise user, I like to use the CentOS. But currently, in our practice, the CentOS support is not ready. So the next very important thing is we need to enable the CentOS deployment. Yeah. As I mentioned during the practice of the bare metal cluster for ICPC, we have found a lot of bugs inside the community project. After that, so our developers have set up that has submitted several bunch of patches to fix them. Maybe all of them has not been merged until now. But I believe after all the patches has been upstreamed into the new release. After that, the bare metal HPC solution will be much better for other users. So I think we have described the solution because of the time it's limited. So there's still a lot of details we have not covered. So if you have some questions, we can talk more. Or if you have some questions, or we can talk more offline. OK. Thank you, Shuichi. Thank you.