 Good morning, good afternoon, good evening, good days. This is Fuchiao from China Mobile giving you this virtual session from Beijing, China. Today, I want to share our continuous integration, continuous testing, and continuous delivery pipeline we're using in China Mobile and also share our view for the DevOps loops for our future telco cloud. So on the second page, I actually listed the contents I want to share in this session. First is about the challenges we see for the telco cloud. And secondly, I want to share the view we have for the whole DevOps loops for future telco cloud. And finally, I want to share some of the experience we got when we were constructing our telco cloud. So talking about NFE, NFE actually the first kind of concept was proposed eight years ago by SCNFE in the white paper. And then SCNFE worked out the framework. It is a very well-known framework for NFE, and it's very simple and straightforward. However, today I want to share with you the real NFE in reality. China Mobile actually launched our telco cloud construction since 2019, which I think is also one of the world's largest NFE projects so far. With tens of thousands of servers in total, we have this cloud located in eight regions across the country. Here I try to give you some of the examples of how our source pool actually looked like. Hope you could have some direct feeling on how the complexity would be for our telco cloud. So you can see, for the first resource pool, we have servers, switches, and distributed storage from all different vendors. However, we have OpenStack and VNFs from the same vendors, which means that interoperability between VNFs and the OpenStack will be much easier because they could have their own private interface. And then, but if you go to the third resource pool, you could see that the VNF and the OpenStack are from different vendors, which means that you should have to deal with the interoperability between these two. And when you go to the second resource pool, you could see some of the VNFs are from the same vendor, but the others are from different vendors. But they use separate orchestrator. But if you go to the fourth one, you could see that they share the same orchestrator. So we have to also deal with the interoperability problem between the orchestrator and the third party VNF manager. So I guess this could probably help you to understand more about how our telco cloud could be like. So on the third page, I want to show the challenges we have. First is about the scalability. We have more than 60 resource pools constructed almost at the same time, which actually require a lot of human powers and resources into this. And each resource pool actually includes physical servers of more than 1,000. And we have actually a huge workload for configuration, deploy, and testing for all these resource pools. And the second challenge, as we see, is multi-vendor problem. I think you have some feeling about that in the previous slides. In total, we have tens of vendors included in our telco cloud. So you could see that we have complicated interoperability work we need to do. And we need to check the interoperability, making sure that they could work together. And when you're actually going deeper into the cross vendor interfaces, you will realize that we are still lacking of a lot of standards and still lacking of a deeper understanding of the standards when you do the interoperability. So issues we see when we actually do in this telco cloud delivery. First is, although we have developing a lot of automation tools in the open source community and also in vendors on product, but when we're actually constructing such a large-scale cloud, we still see that we are lacking of automation tools. Most of the vendors are still deploying the cloud manually or just to use some simple automation scripts. Efficiency is the biggest problem for us. I know automated mission tools, they are developing, but they have never tried these out in such huge scale. So people are not quite so confident with that. And eventually, they decided not to use these automation tools. So the second one is about interoperability. This kind of issue actually happened when different vendors put a need to integrate and work together. And a lot of interoperation needs to be decided and worked out. And even the software needs to be modified on site, which it takes a lot of time and you cannot count on those things that actually are worked out just on site. And third is about the manual configuration difference. When you finish the deployment of the cloud, you need to do a lot of configuration work. However, these configuration may be different in different roles because they are done by different engineers. And some configuration may cause the performance decrease due to the lack of experience of the engineers. So this is also some problem we face in our technical cloud. So now I would like to go to the second part of my talk. It's about our view for the future DevOps loops. DevOps is a very hard topic actually in these years, especially when we talk about cloud native, when we talk about the CI CD for the software. But we also consider that DevOps is very important for our technical cloud. But probably we need to make it much clear how DevOps actually means for telco cloud. It's probably a little bit different from the whole DevOps loops for the software development. For cloud-based telco cloud, we consider it's very important to build up this DevOps loop to make sure that we have a normalized planning, deploy, and feedback look happening actually in an automatic way. It is also important to keep in mind that the scalability and multi-vendor nature of this telco cloud. So when you design the DevOps loops for telco, you need to realize these two things. There are actually a lot of DevOps tools we could already utilize in telco. However, there is also need to develop new tools and even standard interfaces in order to solve the challenge for the scalability and the multi-vendor problems that I mentioned. So on page A, they actually share how we think the DevOps loops could look like for future telco cloud. Within Channel Mobile, we think that it should be down in a continuous way. We call it as continuous integration, continuous testing, and continuous delivery. So in this kind of continuous delivery cycle, we hope it could help us to increase the efficiency of the future network cloud delivery and also help to quickly feedback to the vendor products. So you could see this DevOps loops is different from the software development DevOps loops because it actually connects with vendors. And it actually first go to the lab and then go to the field to go to the delivery site. So it could be a little bit different. However, there are a lot of things we could reuse, we could utilize from the whole DevOps communities. On page nine, I want to share some of the details of how we see this. So first, our CI, CT, CD, it actually includes three phases. We think from CI to CT, we hope all the interoperability issues could be solved. We provide lab environment and CI, CT, automation pipeline to make sure that efficient cross-vendor interoperability actually could happen before we directly go to the delivery on site. So any new versions that actually happened in the vendors could test and iterate within days in our labs. And now we have multi-vendors to actually join this kind of activity to make sure that any time they have their version updates, they could promote that to our version gateway in our lab. And then we will automatically build up the environment, deploy the new version, and do the interoperability test with other vendors' product. So with this, we hope that we could achieve the goal as any vendor match can be tested in advance. Any product can iterate within days. And any lab resources can make full use of. We don't need to actually deploy some software and wait in the lab to do the integration work. Any time you need to do any match, they will deploy automatically in hours. And for the second phase from CT to CD, we hope it could be a precise copy from lab to site. Our versions tested in lab eventually will result in some deployment and configuration scripts and codes. And we hope that these could be precisely copied on site and delivered automatically. So you see, when we do the continuous testing within lab, we actually have a lot of things. And they are all in Docker images. We have the software version that actually passed the test. We have the script for cross vendor interoperability. We have script for automation configuration tools for automatic deployment. A lot of these things, we put them in Docker images. And then we transfer to our site to do the continuous delivery. Mostly the things we need to do on site is we have to modify the configuration parameters according to the scale, and then the whole resource pool could be bring up automatically. We hope that automatic delivery based on image and scripts already tested in lab, the cross vendor interoperability solutions already tested in lab, and the configuration scripts could copy across different pools to avoid the human engineer experience difference for that. So these are mostly the view we have for the future DevOps loops for our telco cloud. And for the third part, I want to share some of the things we are doing in China Mobile. So first, I want to give you some big view of how our network look like. We start since 2019. And now we finish two phases for the construction. We have tens of thousands of physical servers included in the cloud. And they are distributed in eight regions across the country, and you know it's a big country. We have about 20 hardware and software suppliers included in the two phases of construction. So when we do all these construction work, we gradually decided that we need to set up an overall continuous delivery process. So we have a closed loop network delivery process we build up. It helps us to do the whole construction work independent of any vendors or integrators. This whole loop is built based on CI, CD, and automation tools to improve the whole efficiency. So you can see that we do the planning, and then we bring all the things to our lab to do the pre-test. And then we have all these things put in the Docker image and bring that to the onsite deployment. And then we will do an automatic check and a verification, and then go to the operation phase. And when we find problems, issues we need to solve in the operation phase, we will still get back to the planning phase so that for the next construction time, we will adjust all these issues in the planning phase. So you see that this loop we're actually building around multiple automation tools. And we're using CI, CD pipelines to make sure that this closed loop could run. On page 14, I first want to give you some of the hardware automation we are doing. It is actually when we are doing this huge scale of cloud construction, we realize that it's very important to improve the efficiency and quality of hardware integration. Since any hardware defects will eventually influence the software to deploy on that, and will cause actually difficulties in sourcing all the issues. So we actually, the things we are doing is we develop an automation tools for the full lifecycle of hardware delivery. You can see on this whole process, the block in red, they are all doing by the automation tools. While the block in white, they are manual work. You can see there are actually quite a few manual work we need to do in this whole process. We will do the resource planning automatically. And then we will do the pre-test of hardware in lab automatically. When all is done, we will inject all the data into our hardware low-level design data sheet. And then at the same time, all the hardware will put on shelf, and the workers will work on the connection of all those lines. And then we could just turn on all the switches and servers, and we will do automatic device configuration and testing. And we'll do automatic network connection check. And if we have some, and then if there are some issues we find through these testing, we will have analyzed the result put out, and the workers will do the rectify if possible. And then eventually when all the issues are solved, we will do the automatic delivery. So you can see that with this kind of automatic delivery process, we actually reduce our construction time by one-third. It actually only take us 20 minutes to configure all the devices, and 80 minutes to finish all the testing on the single pool of more than 1,000 physical nodes. So the major challenges of all these process we're dealing with is actually there are all different kinds of interfaces from all different vendors' devices. People who understand about hardware probably know the kind of challenges we have. So we have to design an open framework to easily adapt all these vendor devices. This is something that we are working. And we are also contributing this to the elephant community now. I will show the details in the following slices. So on page 15, there are actually some detailed numbers I want to share with you. We have more than 15,000 issues actually found and solved during our construction. And these two figures actually give you some details. We are comparing different solutions. For the manual solutions, it's like how all these work are doing traditionally. All the work configuration are down manually. And then people will do a spot check, which they choose only 5% of the hardware to do the check. However, even that will cost us like 60 days, almost two months to finish all the hardware integration. For solution C, it is actually something we are doing in early phase of our construction. We're still using the manual configuration, but we're doing some automatic testing. And then we will rectify the whole things manually. For solution B, we are actually doing the manual configuration, but we are doing the automatic testing here and automatic rectify. And solution A is the best, all the things are automatically down. So you can see that the time taken for these different solutions will do a comparison of all these things. And you can see solution A actually gives only five days to do all these automatic work. Another thing you need to consider is also the configuration fault rate. When you do all the configuration manually, the fault rate is about 30%. So when we realized this, we decided that it's very important that we also need to do some automatic configuration and also do some automatic rectify. So for solution A and B, we actually do the automatic configuration, which if you check all the codes, the configuration won't be costing any fault rate. And also when we do some rectification, we actually, when we find there are some issues, we will do the automatic configuration to directly change the fault issue. So you won't see that fault issue appear in the following result analysis. So this actually helps us to hugely reduce the fault rate. The best work we actually do is for solution A that we see zero fault rate for the hardware configuration. On page 16, I also want to share with you the software automation delivery loop we're doing. We're using Jenkins, this open source software, to build up the whole CI CD pipeline. And then we are providing a general port description file for both vendor products and CMCC tools. So you can see that all the tools from vendors and from CMCC, they are put into the same Jenkins pipeline to work with each other. And all the tools, they understand a general port description file. This makes tools from different vendors and suppliers that could work together to finish the whole procedure. So when we have a new version upgrade to our gateway, we will use our automatic tools to help do the hardware configuration and environment preparing. And then we will use vendor supplier's software to do the Veeam building and distributed storage deployment, and also do the interoperability work between them using automatic scripts. And then we will use our automation tools to do the test and eventually give them some report and feedback. But like you said, all these things, they are down under a general CI CD pipeline and a general port description files. Page 17 actually shares some of the results we got for the CI CD pipeline. We now actually are connecting with different vendors' labs, and any version updates from vendors will invoke an automatic deploy and assistant check loop in China mobile slab. So with this one, we are able to continue to deploy and test a vendor open stack for more than 10 times a week. And each one actually only cost us less than five hours. So it actually helps us to iterate very fast and precise. Here I also share some of the numbers we have for different vendors. You can see some of the vendors, they could even iterate less than two or three hours. So my last page is I want to share with you some of the contributions we are actually doing to elephant community. So those things we are doing for CI CD, for the DevOps loops within China mobile style code loop, we also see that if we want to build up this loop, we need vendors' help, we need improvements in the whole community for the automation. So we decided that we need to closely work with open source communities to contribute these back to open source and make sure people understand the value here and also develop their own tools based on this. So for the hardware delivery work, which I mentioned previously, we're actually contributing this to the OPNV CRV project. So this HDV tools actually provide an open framework for hardware check. And with quite easy configuration files so that you could adapt different vendor's devices very easily. And you could find some useful link in my slides. And also about the poor description file. This is actually some saying we already down in OPNV for the past few years, but now we are contributing our general description files to PDF2.0, which means PDF2.0 actually need to evolve to have more illustration for the telco cloud to have more details included. So we're actually helping to contribute the general description files we are using within labs to the PDF2.0. So I also provide some useful link in my slides. And finally that's all. If you are interested in contributing our OPNV work, please also feel free to contact our developers. Thank you and hope you enjoy your time in this virtual event. Bye-bye.