 Hello, good afternoon, everyone. Welcome to my presentation. I come from China Mobile. Today, I will give you some introduction about NFV integration test without analysis in China Mobile's Nominet project. You mean, first, this is a gender today. First, the background. And the second cloud platform test and analysis. Last, I will introduce the integration issues and analysis. As you know, Nominet is a future network brand of China Mobile. It describes the vision and the core requirements of China Mobile's future network development. Now, we build the test project aims to unite the industry power and promote Nominet rapid maturity and development. So as you know, the word NOVA and NET, this is NOVA. NOVA include the new architect. We have the network application and the cloud platform. This is our new architect. And then, the next operation, it includes the central control and the dynamic schedule. Last one is new service. We will, based on our network, we will provide the new services, open, ASL, and on demand. This is our NOVA. NOVA, based on our network, will have mobile net, IP net, and the transnet, also the data center network. This is the name of our Nominet. Above this, we have the co-architect, interconnect data center based on telecom integrated cloud. And the way the co-tech is NFA SDN, this is our Nominet network. This is a non-net architect. We have the code tick and the S tick. Above is, this is a code tick. We have, maybe, we have two or three code ticks in one province. And we have many S ticks. At the bottom, we have the AP access point. At AP access point, we have different types. For example, this one, this is a virtualized machine with a dedicated machine. For example, OLT, TTN, OTN, such kind of traditional device. We also have the total dedicated device. And we have some AP with x86 hardware and the traditional telecom device. Above this, we deploy the S tick. As you know, most of our traffic flow, forward traffic and the data traffic in the S tick, which is near to customers, this is a code tick. Mostly, code tick will use the control plane data. So this is a straight layer from the AP S tick to code tick. So now I will introduce the tick. As I mentioned in last slide, tick is a very important concept during our architect. In fact, tick is the basic component of the telecom cloud. And it is standardized micro-module and key infrastructure that meets carrier-grade requirements and can host a variety of virtualized telecom software applications. Tick is the principle to build easily, to quickly copy and deploy with two standards and one unified. So in this, this is the architect. We have two standard. One is the standard networking. We have more straight network definition and isolation based on telecom standards. This is our core networking. And also, we have standardized infrastructure. In this structure, first, we include common x86 hardware. And also, we will have a unified cloud operating system, for example, OpenStack cloud computing. And also, we have a unified management system. As you know, we have the NFV August Treater, OpenL or ONAP. And the SDN's August Treater controls as a unified management. We also have cloud-based resource management with career-class enhancement. So this is two standard and one unified management system. Why we know the tick is as a new data center? Because it has different management model, standardization, computation, replication. There are distributed deployment and networking requirements. We have special requirements in the acceleration, reliability, and so on. This is our tick concept. OK, as in our total architect, we may do the tick integration testing and test the model. And this is the test content. This is the architect of our non-net. At the bottom is hardware. This is SDN controller. This is WIM and HyperWare. About that, we will run every VNFs. And we will do the August Treater. We will auger treat it by the August Treater. Besides this component, we also have some interface. For example, this one is a standard interface. It is the API interface. This one, this one, and this one. Most of them, for example, OpenStack API interface. We also have a standard platform. For example, VNF will run the standard platform, the cloud platform. And also, we have the plug-in. SDN controller will provide the plug-in for the cloud platform. In our tick integration testing, this is the test content. Include hardware, WIM system, SDN, manual, and many kinds of VNF business. So today, I will give you some introduction about all this integration testing and the test route and some analysis. In our normal net project, we have several cloud vendors who provide the OpenStack solutions. And first, we do some tests about the OpenStack solutions. And then I will give you some analysis about this test route. This is a test route. At the left, this item, first, one is the interface test. We use Tempest to test the OpenStack standard interface. In our test, most vendors pass this test. And two is function test. For example, the first one is the WIM function test. Include, as for the function test, include some features which is related with the NFV requirements. For example, the SRV spot, DPTK spot, and the hot migration, such kind of WIM function test. We also have a virtual network function test. This will test the performance. For example, the DPTK acceleration, SRV acceleration, and the queue-in queue spot, and the network card are tested online. We test many kinds of networking function tests. Yes, in this test, maybe some different vendors have different results. This is an average result. We meet some problems. For example, some vendors did not spot queue-in queue, and some not spot DPTK. Yes, next one is the virtual resource schedule test. In this test, we test some schedule policy for the virtual resource management. For example, the anti-affinity features are the heat-template test. All the vendors passed these test scenarios. The third one is the performance test. I will give you the detailed introduction about performance test next. The fourth is the cluster expanded test. Yes, in this test, we only focus on the compute node extension test. All the vendors passed this test. Five years operation and management ability test. For example, we test the virtualization version management. In this test, we include the version update and the go-back. This test, 77% passed. That means some vendors not passed these test scenarios. And the next is the hardware management test. As you know, in telecom, in FV scenarios, there are lots of requirements about the hardware management. So we do some tests about this function. Only half, only 50% passed. Some vendors only spot the hardware turn on and turn off. Such simple functions. The third one is monitoring and alarm tests. Also, the 77% passed. Six is reliability test. We include the hardware failure test. For example, in our OpenStack solutions, in the OpenStack architect, we include the compute node, controller node, also some storage node. So we do the fault test for every scenarios, not only the node, but also about different networks, admin network, application network, and storage network. All this about the hardware failure. Mostly, the vendors passed this test. And the next one is the WIM error recovery test. And the last one is the system backup test. All vendors passed this test. OK, this is the test throughout. Then I will give you the overall analysis. Currently, most of the OpenStack solutions meet the needs of NFV business, but need to improve the maturity and need to further verification, combined with VF. This is our conclusion. First is about OpenStack interface. Temp test was used to test more than one sovereign interface is passed the test, meet the standard interface requirements. And secondly is function test. Now the vendor's OpenStack solutions spot NFV requirements, such as CPU, call bonding, NUMA, hot migration, virtual machine Q&Q, network QS, DPDK, SROV, and so on, such kind of networking acceleration functions will test. And most of the vendors spot such kind of features which has close relationship with NFV. Third one is operating management capacity. In this test, open source product capacity is obviously insufficient. Commercial products, commercial OpenStack solutions have SNMP just for interface spot, but alarm performance statistic data and other information still need to further verification, combined with NFV business. In our next phase test, we will combine VF test to do more verification about the operation management. The fourth is reliability test. VM platform can provide high availability solutions, such as three kinds of nodes fault handling, three kinds of network fault handling, system backup, system process failure self-healing, virtual machine fault handling. But open source products maturity needs to be improved in the reliability. The next step, we will consider both the platform high availability and VFs high availability to form a reasonable solution for VF. As you know, HA solutions is not only dependent on the platform HA solutions, but also it has some close relationship with VF. VF also have some reliability solutions. So we should combine both of them to spot our applications. This is a promise test in the cloud platform. About is the conclusion. From the platform side, the use of DPDK, SRV, NUMA, and other technologies significantly enhance the network to forwarding performance. But the follow-up needs to be combined with business scenarios to further verify. Now we only verify the performance. For example, some acceleration features for the cloud platform. But where it can spot VF business needs, it still needs to further verification. In our next phase, this is the test result. First one is VM DPDK plus cloud SRV. That means VM will enable DPDK program. And in cloud, we will use an SRV compared with, this is compared with scenarios. VM, DPDK, and the cloud only use OS. The package size is, first one is a small size, 64-byte. And the big size is 512-byte. For small size, it has three to six times improvement. Depends on different Wanderer's solutions, they have different improvement times. And for the bandwidth ratio, it's about 50% to 96%. About the big size, the promise enhancement doubled. And the bandwidth ratio is 100%. Yes, this is a summary. Combined SRV and DPDK, we can get better network performance. The second one, we test DPDK. Test scenario is DPDK, both VM and OS. That means the OS DPDK we used in this scenario. And the compared scenario is no DPDK, both VM and OS. So for the small and the big package size, we can get the high performance improvement. For the small package size, we get the promise enhancement about 10 to 12 times. And the big ones was about eight times. Yes, the bandwidth first one, the small one is 35%. And the big one is 50% to 80%. This is a summary. With DPDK technology, the networking forwarding performance improved significantly. The last one, we do some comparison with the NUMA. Test scenario is the same NUMA, DPDK, both VM and OS. The compared one is different NUMA, DPDK, both VM and OS. At this scenario, for the small package size, it's about two times improvement. And for the big package size, it's about doubled. So we have the conclusion NUMA have some help in improving performance. So at the bottom is some comments. One, not only platform network performance optimization, VMF scenarios should be considered to select the appropriate network optimization technologist. That means in platform, we provide many kinds of promise acceleration measures. But which one VMF choose? VMF should take account about it and choose the proper network optimization technologist. The second one is the same DPDK technology, different network problems between wonders. You may some wonders use the DPDK, can get the high problems, but others is less than them. So the software problems enhancement technology need continuous optimization. After this phase, we also do some VMF test. But we only do some test about function test and some management test, not include the promise test. Yes, we use CPE-PC-E-Bound such VMF scenarios from several different wonders. And during this process, because we choose different hardware, different over stack solutions, and now we use different VMF scenarios in our total architect. So we meet lots of integration problems and issues. So next one, I will give you the integration result and do some analysis. This is the architect. When we do the deployment, these three VMF scenarios on our cloud platform, we meet some integration issues. This is a summary of integration issues. We have found 39 issues during integration on the major ones. The major ones is as below. First, this is the item, this is sub-item, and this is some description. First one is the adoption between VMF and the cloud platform. At this item, we find 22 issues. The sub-item is about network issue and software issue. As for network issues, we find MTU configuration issues. For example, in NFC applications, most of the network packages have a big package. So they need the big MTU settings. But so this should have some adoption with the cloud platform and the VMFs. Also, we meet some issues about the Q&Q spot. Some VMF needs Q&Q spot, but some platform have no such functions. And the second one is software issues. These issues, many issues come from this item. For example, the OS version, the VMF pass, pass for the cloud form, and API interface adoption issues. For example, they use different version of OpenStack interface and the heat template adoption and the King Store adoption issues. Of course, we still have other adoption configuration issues. Next one, this item is about VMF. It's not designed according to the cloud mode. At this item, we find 10 issues. This is sub-items. First one is not in accordance with multi-tenant model design. For example, you may see the description about it. First one, VMFM needs to call admin URL with tenant authority. Some VMF is designed to call admin URL with tenant authority. And the next one is some VMF needs layer 2 connection with cloud platform, not layer 3. Because as you know, for our cloud platform, we only provide the IP address. You can't connect to my cloud with layer 2. The next one is no-meet-cloud-platform-network definition to derive. This is some description. Some VMFs communication need to close the security group Some VMFs is connected with outside device by L2, but not L3. VMFs load balance can use the cloud platform's LBAR service. And the last one is generating multi-pots for one VM in the same subnet. For example, we see a VM to create multi-pots, but multi-pots only in the same subnet. This is not spot, normally, this is not spot by OpenSEQ. Maybe you can do it at the bottom. The third one is no sufficiently use of the cloud platform's concept. For example, maybe some VMF can use easy concept, but they can't use it. The last one is not in accordance with cloud platform handling material when derived. For example, every OpenSEQ solution has its features. In our test, in one cloud platform, the auto-recovery failure function will restore VM to the state of the start. So if no user shared disk uses a local disk, maybe the risk of the data. So such kind of features, when we design the VMF, it should notice such kind of features. OK, this is some discussion about the integration issues. This is a conclusion. We have the sessions in the following areas to promote the NFV business in the material. First one is the recommendation that VMFs design in accordance with the cloud model. Currently, most of the VMFs design not fully meet the cloud model based on our test. They only do the deployment from specialized equipment to the virtual machine with little changes. So some way of not take full advantage of the characteristics of the cloud platform to derive. So that means when for the efficient from the traditional model to the standard model, we should do the cloud native derive, make full use of the characteristics of the cloud platform. The second one is the recommendation that the FV-HA design combination with cloud platform-HA. Currently, most of the VMFs have their own reliability. Typically, the reliability of the dual backup design with heartbeat, link, reductance, and so on. Also, they may use the load balance. VMF talent need to take into account the high availability of the platform. The third one is the recommendation to more research about VMF promise. As you know, SROV, DBDK, such kind of promise acceleration technology. But which one is more better for VMF? I think the VMF design should take consideration about it. For some VMF, no use promise acceleration technologies. Through the software distributed processing, also can meet the requirements of VMF. So traditionally, because as you know, the telecom device, they will use the dedicated hardware. So they may do the promise enhancement by the cortical from the hardware, the platform, and the software. But now, you use the third mode. That means multi-VFs will serve the hardware, serve the cloud platform. So you should do the new design to use the features of the cloud features. You can't depend on the hardware or cloud itself. You should make sure you can do some promise enhancement reliability by yourself. Not only depend on the hardware features. OK, that's my presentation. Thank you. Any questions? OK, I have one question. Have you compared the performance of the DPDK against the SRIOV? I mean, do you have figures, packet per second, or? Yes, in fact, we have ever do some comparison with SRIOV and the DPDK. Mostly, as for our task without, DPDK's promise is about half of the SRIOV's promise. So like a strategy from China Mobile for the workloads that have the high throughput like gateways, you will proceed with SRIOV, right? There is no plan to adopt DPDK or maybe FDIO for such workloads? No, because this test route will only test the open-source solution itself, not with the VF workload. So we only test the platform. In fact, we prefer the DPDK such kind of software acceleration because, as you know, SRIOV such kind of hardware using have some limited limitation. So as I just recommendation about it, we stressed VF design to do the new design, make full use of software acceleration technologies, not only depend on the hardware itself. OK, thank you. My last question is that, does China Mobile use this kind of test to do some short listing of vendors or it's not used for this sake? Do you do short listing of vendors? For example, if you have one vendor that fails in many scenarios, didn't fulfill certain performance, does China Mobile take him out of the list or it's just performance testing? As you know, in our total network non-net project, we involve many vendors to do the test. For example, hardware will have many vendors, cloud platform will have many vendors, and VF will also have some vendors. Now we only do the lab test and some PLC test in our pro-wise production site. We may do some evaluation about the total architect and make some standard for the three-layer decoupling. OK, thank you. Yeah. Any other questions? OK, thank you very much.