 Live from San Diego, California, it's theCUBE. Covering KubeCon and CloudNativeCon, brought to you by Red Hat, the CloudNative Computing Foundation and its ecosystem partners. Welcome back to San Diego. It's KubeCon, CloudNativeCon 2019. You're watching theCUBE. I'm Stu Minim, my co-host for three days of live coverage is John Troyer and happy to welcome Fresh off the keynote stage. To my right is Azhar Syed, who's the chief architect for Telco at Red Hat, and the man that was behind the scenes for a lot of it, Han and Garcia, Telco Solutions Manager at Red Hat. Gentlemen, thanks so much for joining us. And thank you very much for inviting us here. Thank you for having us. Interesting keynote, so 5G, my background's networking, we all watch it. Let's say my Telco provider already says that I have something related to 5G on my phone that we grumble a little bit about, but we're not going to talk about that. What we are going to talk about is the keynote. We had China Mobile up on stage. Maybe, I love a little bit behind the scenes, as you were saying, CloudNative enabled not just the keynote and what it's living, but gives a little bit of what happened. Sure, look, when we took on this particular project to build a CloudNative environment for 5G, we spent a lot of time planning. And in fact, this is the guy who actually did most of that work to do a lot of planning in terms of picking different components and getting that together. One of the things that CloudNative environment allows us to do is bring things up quickly, the resilience part of it, and the scale part of it. Those are the two important components and attributes of CloudNative. In fact, what happened last night was obviously one of the circuit breakers tripped and we actually lost power to that particular entire part that you saw on stage. I mean, nobody knows about this. I didn't talk about it as part of the keynote, but guess what? Through, because it was CloudNative, because it was built in an automated fashion, people were able to work. Yes, they spent about three hours or so to actually get that back up, but we got it back up and running and we showed it live today. But I'm not trying to stress on how it failed, or why it failed. I'm trying to stress on how quickly things came back up and more importantly, the only CloudNative way of doing things could have done that. Otherwise, it wouldn't have been possible. All right, so Hananez, as the man behind the scenes there, it's great when we have, here's actually the largest telco provider in the world, showing what it's happened. So the title, Kubernetes Everywhere, the telco edge, gives a little bit of a behind the scenes as to kind of the mission of building the solution and how you got your customers, your partners, engaged and excited to participate in this. Well, it was a very interesting enterprise to realize, actually. We took four months, around 15 partners and I would say partners, because in that case, I'm taking Bel-Canada and China Mobile as a partners. They are part of the project. They were giving us a requirement, helping us all the way through it. And together, other more commercial partners and of course, as well as our alliance, like the team in Eurocome and Open Interface Alliance is well working with us. It was about 80 to 100 people working behind the scenes to get this work, to have a lab directly completely set up with a full 4G containerized mobile network in France. I have the same in Montreal, 4G and 5G course, directly in Montreal as well, in one of our partners, Calum Labs. And then bringing here the 5G pop and have everything connected through the public cloud. So we have everything in there. So all the technology, all the mobile technology was there. We have enterprise technology that we're using to connect all the labs and the pop here with the public cloud through SD1 technology. And we have, of course, deployed, as was mentioned, we deployed Kubernetes on the public cloud and we have, as well, Kubernetes, Rehat OpenShift container platform running on all the prems in the lab in France, the lab in Montreal and the pop here. As I say, it was kind of an interesting enterprise. We have some hiccups last night, but we were able to put that out. The world of telco, very specialized, very high service level agreements. I always want my phone to work. And so a little bit uses different terminology than the rest of IT sometimes, right? That's true, yes. NFV and VNF and VCO. But so maybe let's tell people a little bit, like, what are we actually talking about here? I mean, people also may not be following Edge and telco and what's actually sitting in their hometown or it used to be embedded chips and then it was like Linux, but we're actually talking about installing Kubernetes clusters in a lot of different, really interesting topologies. That's absolutely true. Actually, with the way Hanan described it was perfect in the sense that we actually had Kubernetes clusters sitting in a data center environment in France, in Montreal, and a remote pop that's sitting here on stage. So it was not just independent clusters, but it was also stretch clusters where we actually had some worker nodes here that were attached back to the Montreal cluster. So the flexibility that it gave us was just awesome. We can't achieve that in Jambo. But you brought up an interesting topic around carrier or the telcos operated in an environment which is different and cloud-native principles are a little bit different, where they want very high availability, they want very high reliability, a good amount of redundancy. Well, cloud-native environment actually provides those attributes to you, but the operational model is very different. You have to almost use code as throwaway, hardware as throwaway, and do a horizontal scale model to be able to build that. Whereas in the older environment, hardware was a premium, switches and routers were a premium, and you couldn't have a failure. So you needed all of those compliance of high availability and upgradability and so on. Here I'm upgrading processes in Linux. I'm upgrading applications. I can go deploy anytime, tear them down anytime. I'm monitoring the infrastructure using metrics, using telemetry. That wasn't the case before. So a different operating environment, but it provides actually better residency models than what telcos are actually used to. It's a complicated ecosystem to put all these pieces together. Gives a little insight as to Red Hat's leadership and the partners that help you put it together. I would let him answer that. I'd say it's not our first rodeo. We have been working on the virtual central office project with the Lino Foundation Networking and OPNFV community for three last years, let's say. And the interesting part of this one is that even though we typically was working with what the technology that they are using now, we decided it's time to go with the technology that we'll be using from now on. But of course, there is a set of partners that we need. We need to build the infrastructure from scratch. So for example, we have Lenovo was bringing all the servers for the setup in Montreal and here in San Diego, which actually the San Diego pub was built originally in Raleigh, in Lenovo facilities and cheap all over the country to here for the show. And then we have the fabric part, so the network part that's this column. This was bringing us the software-defined fabric to connect all the infrastructure. And then we start building the software layers on top. So we have Rehab OpenShift Container Platform for the two, completely deployed on the metal servers. And then we start adding all the rest of the component like the 4G core from Ultran, like the 4G and 5G radio from Ultran, together with Intel, ComScope. That is building, start building the mobile part of it. In Montreal, in San Diego. And then we add on top of that, then we start adding the IMS core in the public cloud. And then we connect everything through the SD1 by tuning. So a couple of things that I'd like to highlight in terms of coordinating partners, getting to know when they're ready. Figuring out an onboarding process that gives them a sandbox to play with their configurations first, before you connect them back into the main environment. Partitioning that, working simultaneously with multiple, we had a Slack board that was full of messages every day. We had a nonstop, you know, every morning we had a scrum call, right, like a scrum meeting. Every morning, just a daily stand up from 8.30 to 9.30. And we continued that. All over the day, actually. So, Azhar, one of the things I really like to China mobile when they talked about in the keynote, first of all, they said, you know the promise, 20 by 2026, you know, it's rainbows and unicorns and 5G will help enable so much around the planet, seriously. But today, Shifu talked about major challenge in the rollout and infrastructure and service and capability, so help us understand a little bit the hype from reality of where we are with 5G, what we could expect the next couple of years. Absolutely, we are going through the hype phase right now, right? We are absolutely, all the operators want a 5G service to be delivered, for sure. The reason why they want it to be delivered is they don't want to be left behind. Now, there are some operators who are moving more opportunistic and looking at 5G as a way to insert themselves into different conversations, IoT conversation, smart city conversation, right? Edge compute conversation. So they're being very strategic about how they pick the set of technologies, how they go deploy in that particular infrastructure and strategically offer capabilities and build partnerships. Nobody is going to rip out their existing 3G, 4G network and replace that with 5G by 2026, it's not going to happen. But what will happen by 2026 is an incremental phase of services that will be continued to offer. As an example, I'll give you, cable providers are looking at 5G as a way to get into homes because they can deploy a millimeter wave band radio closer to the house and get a very high speed, multi-gigabit high speed connection into the home without having to worry about what's your copper look like? Do I have fiber to the home? Do I have fiber to the business? And so on and so forth. So that's actually an interesting way. Okay, so you're solving the last mile issue in a very targeted use case. Absolutely, so that's one. The other area might be running a partnership with BMW, Toyota and some of these car companies to provide telemetry back from cars into their own operating environment so that they know what's going on, what's being used, how's it being used, how can we do pro-diagnosis before the car actually begins to fail? Pick private environments like oil and gas, mining. They're going to go deploy public safety and security where all of these policemen and safety personnel are required to now use body camps. Now you have video feeds coming from hundreds of people that are deployed on incidents. Now you can take that information, you need high speed broadband, you need the ability to analyze data and do analytics and provide feedback immediately so that they can actually act. So literally this specific targeted use case, even a country like India, where they're talking about using 5G for very specific use cases, not replacing your phone calling. I love that point and it kind of ties back into some of the other things you were saying about the agility in the operational model and I relate it back to IT. Again, my perception of some telco may be 20 years old and that they had a tendency to do very monolithic projects when you're rolling out infrastructure across a country, there's a certain monolithic nature to it but you're talking about rolling out one, rolling out individual projects, rolling out, that's also the advice we give to IT, try it with one thing, try OpenShift with one application. Absolutely, absolutely. And then also though, it takes the upskilling and the cultural model. So, with your telco managers who were on Slack there with you and I don't know if there's any relation, any other kind of things to pull out about the mirror of the IT transformation with telco transformation and culture. That's actually a good point that you bring up, right? Look, the cost of building a 5G infrastructure from ground up is extremely high. If they want to completely revamp that, you're talking about replacing every single radio, you're talking about adding capacity, you're talking about adding, you know, backhaul capacity and so on. So, that isn't going to happen overnight, it's going to happen, it may take even more 10 years. I mean, the most interesting thing that I saw was even LTE is going to grow, LTE subscriber count is going to grow for the next two years before it flattens. So, we're talking about LTE 4G that's been around for a decade almost, right? And it's going to still grow for the next two years, then it's going to flatten and then you'll start to see more 5G subscribers. Now, back to the point that you were bringing up in terms of operational model change and in terms of how things would be IT principles, applying IT principles to telco. There are still some challenges that we need to solve in Kubernetes environment in particular to address the telco side of the house. And in fact, through this particular proof of concept, that was one of the things we were really attempting to highlight and shine a light on. But in terms of operational models, what's used applicable in IT will now be totally applicable on the telco network. The CICD pipeline, the delivery of applications and software, the testing and integration methodology, the operational models, absolutely those. In fact, I actually have a number of service providers of telcos that I talked to who are actually thinking about a common platform for IT and telco network. And they are now saying, okay, Red Hat, can you help us in terms of designing this type of a system? So I think what Hanan could speak to you a little bit about in this context is how the same infrastructure can be used for any kind of application. So you want to talk about how the Kubernetes platform can be used to deploy CNFs and then to deploy applications and how you've shown that. Yeah, well, this is what we have been doing, right? So we have the Kubernetes platform that is actually deploying the services. We have all these partners that are bringing their cloud-native applications on top of that, that what we are calling the CNF, the cloud-native network functions. And basically what we were doing as well during the whole process is that we have those partners that are still developing, they're still finishing the software. So we were building and deploying at the same time and testing at the same time during the last four months. And even, I can tell you, even just to the night. Even last night. So the full CI-CD pipeline that we deploy in IT side, here it is in operation on the network side. Well, yeah, so I want to give you the final word because John was talking about IT cycles. You know, if you think about enterprises, how long they used to take to deploy things and what cloud data is doing for them, it sounds like we're going through a similar transformation at Telcos. Absolutely, in a big way. Telcos are actually deploying private cloud environment and they're also leveraging public cloud environment. In fact, sometimes they're using public cloud as a sandbox for their development to be completed until they get deployed in a private cloud environment. They still need the private cloud environment for their own purposes like security, data sovereignty, and their own operational needs. So, but they want to make it as transparent as possible. And in fact, that was one of the things we want to also attempt it to show which is a public cloud today, a private cloud and bare metal, a private cloud and open stack. And it was like, and you know, it came together, it worked, but it is real, that's more important. And for enterprise and for Telcos to be literally going down the same path with respect to their applications, their services, and their operational models, I think this is really a dream come true. Well, congratulations on the demo, but even more importantly, congratulations on the progress. Great to see the global impact this is going to have in the telecommunications market. Definitely look forward to hearing more in the future. Thank you very much. Thank you for the opportunity to actually be here. All right, for John Troyer, I'm Stu Miniman, back with lots more here from KubeCon, CloudNativeCon 2019 in San Diego, California. Thanks for watching theCUBE.