 Hello, good morning, good afternoon. I come from China Mobile. In this presentation, I will introduce you some test framework proposal in China Mobile, non-net project. First, this is the agenda of my presentation. First, I will introduce the background. And then, I will introduce some non-net project in China Mobile. I don't know where you know the non-net project. In fact, it is an NFV project in China Mobile. The third one I will introduce is the most important thing is our automatic integration verification framework. First is about the background. As you know, if we mentioned the test framework, first, we should take a talk about the evolution about the telecom development architect. At the left side, it is a traditional deployment architect. You may see many VNF applications deployed on the dedicated platform. This is a dedicated hardware. But the mode is changed from the vertical mode to the third mode. What means third mode? That means we will provide the common hardware, the x86 hardware. And based on the hardware, we will provide the common cloud platform. And above this, we will use the virtualization technologies, deployment, every kind of VNF applications. For example, the Voity, IMS, IMS, and the EPC, and others. So this is a change about telecom deployment. As you know, in ABAU architect, also the test method have evolution. ABAU is a traditional test. This is our traditional telecom device. If we want to test this device, we should have lots of around the device to the integration test. During this test, most of all, we will depend on our vendors who provide the device. So the test method has evolution. This is now in the cloud, in the third mode platform. How to do the test? This is NFV, Decovering Test Architect. As you know, this is our cloud platform. We will test all the VNF, which is provided by different vendors. So you may see the test method is different from the traditional to the current. So next one, I will introduce our practice in China Mobile. How to do the test in our NFV project, NodeNet project. Mostly, we will do two jobs. One is deployment, another is test. This is the architect. In this architect, I will introduce the process, what we have done during our NodeNet project. This is the architect of ETSI reference. At the bottom is the X86 hardware. Above right is TIC, TIC is Telecom Integrated Cloud Platform, which means the WIM and HyperWire. This is the SDIN controller. And this is the WIMF drawn on the TIC platform. And this is the manual orchestrator. Except these components, we still have many interfaces, three kind of interface. First one is the red one is the interface, which means some API interface, this red one. And the second one, the green one is the standard platform. For example, every WIMF means the virtual machine. And we'll deploy it on the TIC cloud platform. And the third one is the plug-in. Plug-in means, for example, every SDIN controller will provide the plug-in in the OpenStack Cloud Platform. OK, this is the architect. In our project, in fact, we do lots of deployment at test. At the right side, we do SAS deployment at test. First is TIC deployment and test. In this item, we do the WIM deployment. And during this process, we should solve some problems. For example, the adoption between the hardware and the cloud. And then we will do the verification. This one, we will do some function test, performance test, reliability test, and option test. After the TIC deployment at test, we should do the SDIN deployment at test. And first, we will do SDIN deployment and do some SDIN controller function test and do the SDIN and NFC combination. After SDIN development and test, we will install many kinds of VNF and do the VNF deployment at test. After that, also, we will do the test about the manual test or the trigger test. This is the whole process in our non-net project. We will do SAS kind of deployment at test. Next one, I will give you the detailed information about our test about every component. This one is TIC deployment and test. As you know, if we want to do the TIC deployment, first, we should do some network architect design. As you know, this is the deployment architect in every OpenStack platform. Maybe different one that provides different OpenStack solutions. But mostly, we should include the SAS kind of architect. For example, this has different node controller node and network node, computer node. Maybe also, you still have some storage nodes which provide the distributed storage. Except this node, we still have some network play. For example, the administrative play and some application network and some storage network and so on. So when we do the deployment, first, we should do the network architect design. And then we should do the server design. Of course, we will, in our all the architect, we include many kinds of nodes, installation nodes, controller nodes, computer nodes, and storage nodes. After that, we still do the HA design. For example, in the controller node, we will provide many OpenStack services. We should provide the AC solutions for these services. After the deployment, we will do the tick test. First, we should do the function test. For example, affinity or anti-affinity and so on. After that, it's about the performance test. As you know, VNF needs high performance. So we use many kinds of performance technologies. For example, SRIV, DPDK, Neuma, and the OAS enhancement, and so on. And we still do some computing elastic and reliability test. In our reliability, we do the test about we emulate some fault. For example, it's a different node fault. We have different node rows. Some node is a controller node. And some one is a computer node. Some one is storage node. We will do the default test on all these nodes. And also, we do a test about a different network fault. And the last one, we will do the operation management. For example, the update and alarm. So these are our tick deployment and test. After that, we will do the architect. As you know, we should do the SDN deployment hand test. This is the SDN architect in our In-Channel Mobiles, In-Channel Mobiles non-net project. This is open O, but now it is combined with AT&T E-com as own app. And this is Auger Tritter globally. These are two ticks, the Cortic and the Aztec. As you know, we have many ticks in China Mobiles non-net project. And we deploy many ticks in different programs. And in our central data center. So these are two ticks, Cortic and Aztec. And mostly, the control plane, this is the flow about the control plane. This is the flow about the forward plane. You may see the tick interaction based on SDN. So first, SDN will do some connection between ticks. For example, no matter between the Cortic and the Aztec. Another one, SDN will do some data network flow in analytics. So as you know, we will do the SDN deployment and SDN test. For SDN deployment, first we should install the SDN controller. And then we will install the SDN gateway. After that, we will, these two components is outside the cloud platform. But these two ones is inside the OpenStack platform. We will change the network from the default design in Neutron to the SDN. We will install SDN plug-in in the controller node. And also, we will do some less update according to the SDN solutions. This is the SDN deployment. And after that, we'll do the SDN test. First, we will do the basic network function test. And then we will do the security group test, bandwidth-limited test, network traffic, statistic, and so on, all kinds of SDN test. This is our SDN deployment and test. After that, we will do the VNF deployment and test. I don't know, this is all the VNF applications we will test in China Mobile. At first, we do a test about NBIOT, which is the EPC, CPE test, E-Bound test, and IMS test. These tools are deployment on our FACE2. You means different VNF on the same tick and orchestrated by the manual. We will test all this VNF. And first, do deployment, and then do test. Next phase, I will give you an example about the VNF deployment and test. I will take the CPE, the VCP deployment and test as an example. This is the architect of CPE. As you know, in CPE, these ones are hardware. They will deploy the deployment outside of our tick, of our platform. And this one will deploy it in our tick platform. This one is deployed on the eighth tick and this manual is deployed on the code tick. But we have the connection between the round device to the device in our tick platform. So as mentioned in the VCP deployment, first, we should do the spherical equipment and the simulator deployment, this kind of device. And the second, we will do the SCP deployment on our cloud platform. Then we will make the connection through network between the brass and the SCP. This is the deployment network. After that, we will do the test. For example, the VNF lifecycle management, business floor classification and marking, a CPE basic function test. For example, in our cloud, this VNF will have different functions, net function, DCP function, or DNS function, or PPOE function, such kind of VNF applications. And also, we will do lots of value-added business tests. This is the VNF deployment test. Although we have many kinds of VNF application tests from many vendors, but the process deployment and test may be the same. OK, after they introduce about the process of our deployment and test, I will give you, you may see, the total process is about from the TIC deployment and test, the SDN deployment and test, VNF deployment and test, manual deployment test. This is the process in our normal net project. We already have done. This is our hardware architect. As you know, in our project, we involved six vendors, OpenSec solution. And this is our data center. We have multi TICs. And we also have multi RECs, which is the other test equipment. For other test equipment, we have TIC test equipment. For example, network test instrument, software test tools, emulation tools. For SDN test, we have some SDN controller gateway. For VNF test, we have some vehicle equipment instrumentation from ESL or Sprint, other emulation tools. All these test equipment deploy on these areas. And also, we should do the interconnection between the TICs and the other device, around the device. So we should do the interconnection. OK, that's all work we have already done. But how about the integration test architect? We will have some deep thinking about it. At the bottom, this is Weihua Dao, the TIC basic environment. And based on the TIC basic environment, as you know, we have to the development configuration and the test on all the components. But how about the integration architect? We do our work, our jobs. We summarize the integration architect. And we do the integration process and find lots of verification tools. All these things consist of our integration architect. But about that, we still have new things. We will build the development architect, include the VNF optimization and the VNF optimization. As you know, the VNF has evolution, generation by generation. So we should build a process to enhance our integration test. So this is the evolution. You may see the NFV process is also a CICD. At the left side, you may see the TIC development, include hardware, WIM, SDN, VNF manual, which consists the ETSI architect. Maybe this component is provided by different vendors. After their development, they will deployment on our platform and do TIC integration verification. For example, many kinds of components. After that, it will be deployed on the production and go to production. In production, we will find lots of problem issues. And they will feedback to the development. So this is a circle, wronged and wronged. So in this process, traditionally, the carrier operator will depend on the vendors. Most of the test, integrated test, is depend on the vendors who provide the VNF, who provide the telecom device. But now, the view is changed. The carrier itself should take the responsibility of the integration. This is the chance, our integration test chance. As you know, next step, we will try open NFV architect as a reference. This is an open NFV reference. We have the Jenkins master and the Jenkins slew. And do the installation. For example, use Kampas Docker and do some tests. Use test tools, for example, from test on the R disk. About that, we still have some customized test case. We will integrate this test case to the framework. And we also will use some verification tools to verify the platform. But because, as you know, the open NFV now, some kind of test is not involved in open NFV architect. We will take it as a reference and do some enhancement to use these open NFV tools. This is our plan. And we will do some tests after in our next phase in non-net product. OK, thank you. Any questions? Yes, hello. Thank you. Presentation. In your plan with the test cycles for the different components of your deployment, the VNFs, the VIM, MANO, and so forth, do you intend to make upgrades into production only for all components in one tested group? Or do you intend to be able to move new versions into production, let's say, for VNF or for SDN control independently? OK. In fact, now we will have to do some tests, not we build the non-net product, which is the future of the, which have the future mission of the tandem mobile's NFV evolution. So this is the first phase. First phase, we will do the test in our lab and in many of, in many programs aside. So just now I introduced the process, what we had done. But I think now we will do the test framework proposal, which will deploy our production. For example, next phase, we will push the VNF NFV to the production. And we plan to use such kind of test framework to the production. But now it is only the lab test. OK. Any other questions? Are you leveraging OpenStack Tempest for any of your verifications? And then have you extended it to do any VNF testing? You mean the VNF test in this circle? Yeah. Are you using Tempest anywhere? Now we will not use Tempest. But as you know, next phase, we will use, oh, sorry. In fact, we have, in fact, in the WIM test, we use for the API interface test, we use Tempest. And you know, different wanders, different OPSAC solutions providers have different tests throughout. As you know, Tempest test case is about more than 100 test cases. And we have to do the tests on different wanders. Mostly what we need in FA applications, the wanders OPSAC solution passed the Tempest test. OK. Yes. Deployment, configure, research, and test. In fact, the test, we contains the basic function test, performance test, reliability test, operation test, such kind of test. But most of our test is focused on the features which we will use in the NFV applications. Six vendors for all NFV ecosystem. These six vendors is only for VNF or are all, for example, for hardware, for VIN, for another? For all kinds of components? All of components. Yes. And another question is, how many VNF vendors have you got? Approximately two, one, three? OK. This is this kind of for VNF. This is NBIOT. For example, NBIOT could be one vendor. CP could be another vendor. You have in your deploy many vendors for VNFs or only one brand? Yes, we have many vendors. For example, we have three vendors provide the NBIOT solutions. And three vendors provide CPE. In fact, most of the vendors want to take part in our non-net project. We choose three vendors involved in our project, do some tests. We think after we integrate a test, we will do the standard integration process. And then other vendors will take part in, participate in our integration and follow the standard integration flow. So you have an integrator or partner to do all the tests or only China mobile do us own? Now, in fact, we only choose some vendors to test, not the fixed vendors. OK, thanks. Here you come. Any other questions? OK, thank you to come here to my presentation. I think after the presentation, if you have some questions, you may contact me. OK, thanks.