 So good afternoon everyone. So just now they're like two gentlemen ask me whether this presentation will be Chinese or English. So don't worry it will be English and I can speak Chinese. So if you have a question you want to ask in Chinese you can also ask me. So call me the best presenter this time. Okay so I'm Fuqiao from Channel Mobile. I guess my topic is not like like edge computing or blockchain or any that hot topic. Sometimes I actually kind of matter their topics but my topic somehow includes like CI a lot of difficult things that we met when we are doing all these NFE things but hope it will interest all of you because this is my real world. So first some introduction about myself. We actually have a re-argue early this year. So now I'm in the what we called as a network cloud continuous delivery center. I mean that center so the name is quite confusing actually sometimes. So I'm now currently leading a developer team in that center and basically we're trying to do some automation for the delivery for our NFE cloud. So this is probably the contents for today. I think that probably I will share first why we as operators begin to pay attention to the integration and delivery which seems like just a problem for vendors beforehand. And secondly is how we actually can improve that. And third is about something that we are actually doing in China Channel Mobile is an auto and automatic delivery tool sets. And fourth one I'm very happy that I eventually have some best practice that I can share with you which we actually have quite good experience bringing up and a huge resource pool within this year. So let's try to begin. So NFE cloud this is actually an old topic. It actually is like I guess it's almost seven years when SC first published the NFE white paper talking about the whole concept here. Operators vendors actually puts a lot of money in this but I have to say it's still quite difficult for NFE to actually deploy. I guess this is the same experience for most of the operators and vendors. So why is that from my team's perspective probably the integration is the most difficult one because currently most of the virtualized clouds that I actually deployed in operators they are like still bonded to some single vendors of software and that actually keeps operators experienced the whole benefits that the NFE technology can bring to them like they cannot enjoy the reduce of cost. They cannot actually enjoy improving of efficiency because they still closely rely on the software vendor provider. So we guess that probably the operators probably need to look inside. If they want to actually enjoy this new technology probably they also need to improve their own capability. I guess this is also something that gradually being realized by a lot of operators like just last month in September. I heard the network VP from AT&T Amy talks give this speech in the Illinois summit in Antwerp and she mentioned about that the NFE cloud is actually growing from virtualized to white box and probably now we are at the process for automation. By automation he mentioned a lot about airship about CNTD. I guess probably the most meaning for that one is actually operators need to improve their actually capability to use this network. So I guess that is also something that we are actually targeting for. Yesterday I also listened to one of the speech from Chatno telecom. They also mentioned about they are working on some ultimate testing tools and I guess that is also for the same approach. So for us we are actually also looking deep into the details of the whole integration of the NFE and hope that we could improve our capability to do so so that we could eventually solve the whole mass of NFE. So NFE integration first I want to make the concept quite clear so that people will not misunderstood about what I'm talking about. So this is just the whole process how you bring up the whole cloud. It's actually almost the same as what we are doing with public and private clouds. So I guess we actually can have lots of experience and lessons learned from those clouds. But also NFE have their own problems like the scalability like the different vendors from Telco. That probably is some difference here. So integration changes actually a lot. Traditionally for operators for telco operators especially integration just means that a huge box bring into their central office. They will just make a few where connected and then it's all over. But now actually this whole thing changed a lot for them. First the scale is actually huge for them taking this year. China mobile is actually building our phase one of our network cloud which includes 20,000 servers, 4,000 switches and the network connection is actually over 180,000 lines. So that work is actually almost impossible for us to actually do that. And it's quite complicated. And also another problem is about multi vendor. Traditionally we actually rely on one single vendor to be as the integrator. And we don't need to worry about anything else. But now when you go to NFE, this time we actually have 10 more vendors. We have different vendors for servers, for switches. We have two different integrators actually happens. And lots of different software vendors as well. So this actually gives us a lot of challenges which is totally different from what we are doing in the past. And we have a lot of requirements and expectations from different departments. Like they would require us to reduce the whole integration time. They would like us to reduce the whole cost to do the integration. And also they would expect to have an overall test for all the lines and the servers and switches for each single module. And also reduce manual efforts. A lot of that requirements. So how can we actually face all these things? And it really makes the telco operators enjoy the whole benefits. So we are thinking about probably the integration for NFE cloud is the key. And where actually we need to look into. We think there are three key components that actually makes this integration not good for now and probably can't improve in the future. One is actually the working flow. Traditionally telco operators is like they care less about this whole working flow. It's something that should be taken care of by the vendors, by the integrators. But now we have multiple vendors to join inside. Probably it's the time that operators need to look into this working flow. So we actually, in the beginning of this year, we actually spent a lot of time talking with different vendors, integrators. Like to see actually, is there anything that we actually can do previous to the whole integration? Whether there's any process we could do simultaneously? So this looking to working flow actually helps allow to improve the whole efficiency. And the second one is automatic tools, which apparently that's very important. And the third one is about unified data. We actually have that kind of experience is that we ask we have one integrated from one vendor vendor a and vendor is will integrate some software from vendor B. So he basically just send a table sheet asking some parameters there. And then vendor B just reply with them a table sheet. But it's actually according to vendor B is a format. So they are just like on and on just debating on how we you actually call this the network or things like that. So probably we think that we need to make some common language between these citizens. So beginning last year, actually, we tried to define some unified data through the whole integration area. So this year, we actually try to use a common data definition for our hardware integration. And it really helps a lot. The good thing is that when we finish the hardware integration, all the software vendors, when they get into this, this resource pool, they don't need actually to check the whole resources again, like before, because they already have the real data here. And it's all in a common language format. So I will go to in details, like for the standard working flow, they are like three main things we actually improve. Why is that we actually paralyzed some of the process like I just mentioned. And then the second thing is important to us is we do some pre configuration. There are lots of things actually the servers, the sweet vendors can do before before they actually ship their equipment to our central office. And these really helps a lot. And they also they have the great willingness to do so because that will save their time to actually do some on site configuration. And so we actually have a lab pre testing, making sure that everything that gets into our central office, we have some tests beforehand, making sure it's all right. Before that. And this actually helps us to improve the whole flow. And second is about what I said about unified data, this low level design, I guess every vendor has has that. But at every vendor has a quite different one from each other. So we guess that this whole data is a soul for the integration. And probably we need to define some common language between these these things. So we try to, you know, we have our like, in Chinese, we call it Shui Ji Yuan, but in English, probably is something like design department in the telco that will actually design the whole things for the resource pool, not that details, but a lot of roles there. So we are doing some automation translating that design doc into low level design, so that the vendors can do all the things accordingly. So you can see that we have this design doc. This is generated by some design department. And we will configure that into the hardware load the LLD and then to the software LLD. And eventually, this is also quite important data source for this. So we have to do the acceptance test. And last is about test tools automation. This one is some somehow I don't think I need to spend too much time talking about that in open stuff summit because people here understand about automation and understand about importance for this. But actually, there are lots of actually open source tools to do this. Yesterday from the China telecoms speech, they also mentioned they are using dovetail and OVP from OVP and fee. I guess that's good news. We are also utilizing all these. We are using Ansible. We are using like 10 pass really a lot of these things and also we are developing our own tools in order to fit into the gap. So then I will use a full session actually to introduce about our tools. Forgive me about the name. We actually are not quite so serious about that at the beginning. But it calls auto. Actually, what we are actually planning to is that we want to figure out whether there are any gaps that actually we are we actually need to do the whole automatic integration through the whole loop. So we look into the vendors tools, we look into open source tools. If we can hardly find any that actually fits in our requirements, we try to develop some of our own tools. So now you can see that the thing that in black ones is something we have already worked out and the gray ones is actually we are working on. And we try to cover the whole part. But I guess we are actually good at doing the exception test because that is the most requirement from our different departments from the headquarters. So we did that one a lot. Is this one? Auto test part. But also we try to do some auto designing like what I just say about translating to the from the designing dog into some real designing part. And also we are trying to do some auto configuration so that vendors do not need to actually go to outside from configurations, we could do that automatically. So after this, the next, the next talk, I will talk about like how we actually use auto pipeline to connect all different components, modules together, and then so that we could provide some automatic integration in this year's network integration. So these tools actually we've already used in some different few testing, like we have our purchasing testing, which requires lots of tests. So we actually do the VNF automatic testing for more than 400 times. And also for we also did a lot of testing for five G field trials. And it really improves our efficiencies a lot. So this one is actually try to cover the whole things about how we actually want the auto to be. First is about the planning. We hope that we could having the unified LD data and have it this one to translate into each different vendors data, hope people could talk with each other. And for hardware integration, we actually have tried that out in this year's, this year's deployment, which is a turnkey effort. And software integration, we actually are doing some PUC with multiple vendors about that. We hope that we could bring out a proof of concept in next year's Mobile World Congress. And last thing is about automatic testing. So about that, this one is actually what I really want to share with you is what we actually expected the future cloud integration should be like. We're thinking that probably traditional way is not efficient enough. So we actually try to convert this ideas into multiple people in multiple communities and summits like this. So we're thinking that we should also borrow the continuous concept to future cloud. Like we could call it as continuous integration, continuous testing and continuous delivery, which means that the components, the software should be integrated in the vendors labs or some develop centers. And then it will go to the integration labs and do the continuous testing. And then if that pass, it can eventually go to our field, our central central office to do the continuous delivery. And we hope that eventually with auto with the whole unified data that we defined with the standardized working flow that we defined, eventually we could reach to this. And finally, here's some of our best practice. So this year, we actually built our NFV cloud is it was, it was, it is actually distributed in eight provinces. And for phase one includes like more than 20,000 servers. So it's a huge one. And we will have phase two like late later this year. And our team has actually been engaged in this whole integration delivery effort in eight provinces since March of this year. And auto is actually using these eight provinces to provide automatic configuration of hardware, which is interesting. So currently, I actually have to update this slide because we just finished the last provinces hardware integration. So it's less counted eight provinces for now. So we actually improved the whole efficiency. At first, the headquarter actually expect us to finish all the whole hardware integration work, including the hardware on the shelf, do the network collection in six months. And we actually try to improve that to four months. But for the last province, because we have enough experience here, we shrink that time to two months. It still probably is too long for the people here, but hope we could improve. So that this speaker actually shows how we do the hardware integration. So first, some preparation work, of course, we need to prepare for the data vendor, provide detailed design data. And also, we have that design document from our design department, and we translate it into our low level design. And then we should have like workers to have the servers, the switches to be ready on the shelf, and then have do the network connections automatically, manually, of course. And then we will install this auto server. And then after all is just all being done by automatic way. Like normally for resource pool of about 900 nodes, that's mostly the scale of our resource pool. One and a half hours is needed to do the automatic configuration of servers and switches. And we also need to do a overall checking. I do not include the detailed checking things that we listed here, but it actually includes already IPMI interface that we will use in the following software integration. It also includes some basic functional testing for the servers and switches. And also we check all the topology connections between the switches and the servers and check if they are due according to the design. And after doing all these checkings, and probably will detect a lot of problems like nick damage, power model damage, a lot of different problems. And then we will do the iteration for correction. So normally this probably will iterate like 20 to 30 times. And the problems actually will shrink and eventually we'll have to final report. So this is actually our first successful experience in Nanjing province. First we're testing on 1000 plus hardware and we have 7000 wears and we do that in one hour and 20 minutes. And the first time we actually detected more than 200 hours, but they are actually be correct in two days. So we actually have some comparison with like human, of course, humans, something like spot checking, you cannot do full scale checking, you cannot hardly promise the occurrence. And also compare with some vendor tools. I cannot see which vendor it is, but actually we have some good advantage over them. And here are some results that we actually get from the eight provinces. We actually find some major arrows here is about network. That's something actually easy to expect. And also because that would do a lot of brief configuration pre test in the lab, we actually avoid a lot of server and switch arrow on site. So most of the problems actually happen at the networking. And this is how things actually iterate in provinces. The above figure actually shows seven provinces problem accumulate how they actually iterate in I guess one week. You can see that the problems are just a shrink. And this one is actually showing something happened in Sichuan province, where the problem actually shrink after in two weeks. So some conclusions and takeaways. So from our perspective, we're thinking that probably NFV integrity is the last one mile problem for NFV. So improving this whole thing's efficiency and quality is quite important for this whole NFV success. And second is operators probably should take some leading roles for the NFV integration in order to improve. We can hardly actually rely on a single vendor to work on that. But we need to collaborate with vendor, of course. After all, it's their software and they are the integration. And standard working flow is actually important for us so that we could do the whole thing efficiently and improve the quantity. And probably operators need to work closer with the integrators and vendors for this standardized flow. And unified data is also quite important thing for the whole thing. And we're also thinking about probably we need to kind of contribute this one to some, some communities so that we could eventually build up a cross vendor unified data. And fifth one is about automatic tools is very important. Like we have already done some exercise here we do and we have we have some experience. And we actually we are open to open source and contribute some of our tools and automatic code to to the open source community. And finally is about the pork that we will do for next year's MWC. I hope that will actually deliver our concept for the continuous integration testing and delivery. And that's all. Any questions here? Yeah, for the hardware for hardware testing, I don't think that there's any open FV tools actually exist. And for the software, we actually utilizing a yardstick before. But then it turns out that our testing specifications way more complicated than we are doing. So we just we use some of the yardstick and the others we just develop our own test framework. Yeah, other questions? Okay, thank you. Thank you for your time.