 All right, I guess we can get started. My name's Toby Ford. I'm from AT&T. And I'm Matt Skolson from Ericsson. We're going to talk today, or an extension of what we were talking at the last summit about, the last summit we introduced NFV. We're going to actually go the next step is talk about what we've done over the last six months in terms of evolving OpenStack to be able to support NFV. And then we're actually going to demonstrate an aspect of how to use it to achieve some level of HA underneath as one subset of an NFV solution. Just as an overview, this is the things we're going to go walk through. As I said, we're going to just talk to a little bit of the basics, try to be a little bit incremental to what I've been already talking about at the keynote or in the previous talks, and then go through some issues that we've had trying to bring new functions into OpenStack around solving for NFV problems. And then we're going to go through a demonstration. Again, for those that haven't heard this already, I'll try to be brief. And hopefully you haven't heard my story about this already. But recently I was playing golf, and then they paired me up with this guy who had just retired. And we were walking along, talking about what we do. And I was talking about the cloud. And he told me, oh, I live in the Washington, DC area. So this guy just recently retired. He had started a company 30 years ago supporting mainframe software. And he'd been very successful with the federal government with this mainframe software. So we got into it, and then he's saying, oh, yeah, the cloud. It's just like mainframes. It's a shared thing. It's all the same. It's just coming back around. And then I did not talk to him for the rest of the round. I was ignoring him. This really irked me, because for me, and when you're in Paris, you're supposed to talk like a revolutionary, I feel like the PC revolution and the Linux revolution and the open source sort of things that have happened since Linux have started have kind of broken this mainframe thing and has brought it almost to its knees. And it's almost gone. So that really irked me. And NFV is really an extension of that concept. It is taking what in the telco world is an equivalent to a mainframe, a vertically integrated box, something as simple as DNS, but done at huge scale. We needed to have a vendor provide DNS for us in a fully integrated solution that only scaled vertically, adding more processors, more line cards, et cetera. This is only one example. Obviously, many of the things that we do, from switches to routers to more sophisticated mobile systems, have been in telcos delivered this way. And obviously, there's a lot of shortcomings to this. It takes a long time to upgrade. The evolution cycle is very, very slow. 18, 24 months are short in our world. So extending this and evolving this has been very difficult. So we want something else. We want something akin to what we've gotten in the PC revolution, a marketplace, an ecosystem that is very extensive in terms of ISVs and startups and such. We want something like the Linux revolution where we can work together, especially on non-differentiated functionality, the sum of which is grown dramatically. When you think of all the telcos and what our needs are, we don't really have a lot that's going to be really unique. We want to make something that is open, extensible, transparent. And so we would much rather it be done in a virtual way, in a scale-out way, in a way that's easily extended. So. Sure. So I think I click it on. So this was also what we said in Atlanta. And I think one of the major reasons, I think, why I believe OpenStack is such a good thing for NFV is that it's truly multi-vendor. A lot of other virtualization management systems are having limitations on what stuff you can kind of plug into it. So really having a multi-vendor cloud ecosystem that you can plug in any equipment, any third-party services, any hypervisor, any guest, whatever it should say, really is the key tool for us. Because looking at the picture that Tobish showed, I think there is a multiple of different type of hardware vendors, et cetera, that needs to come into the same arena and play together. We also talked about a lot about the reliability. We are talking about five nines. We are talking about services like mobile telephony requiring five nines on the service level. Five nines is less than five minutes of downtime per year, including planned. That is quite hard requirements. These things we need to keep. We need to keep the SLAs on the service level. But of course, the implementation will be very, very different when you're talking about a cloud infrastructure. But we need to move these applications over to a cloud deployment environment. And we need to retain these characteristics. And I think a lot of the things that we are actually doing now is trying to figure out exactly how can we retain these characteristics on the service level. We also went into some of the examples in Atlanta talking about resource allocation, resource isolation, optimization. This is really a need. I mean, you need to be able to exclusively add cores to a specific VM or exclusively add network capacity to a specific VM. If you take a packet core application, for instance, that will actually require the whole IU capacity to that core in order to work in a good way. We talked about the networking. I think the networking is really an area where I think there is the biggest need, so to say, to improve and do additions. I think one thing with the network functions is that you need to be able to orchestrate the wide area network, not only the local area network within the data center. You need to be able to orchestrate the network that is also outside the data center. Real-time response, if something happens, interrupts the whole area around the virtual switch. Needs to be line rate to the VM. High availability, we talked about five nines. And five nines is really also, I mean, it's very much about fault mitigation, monitoring. A lot of these things that maybe it doesn't sound so much of high availability, but in our view, is the things that we really need to focus on. We need to reduce the risk of single point of failure. Then I think I was reading a lot of the journalists from when we had this panel yesterday, that I think there is a lot of good discussions now about upgrades and backup and restore. I think if you're looking on telecom, we should really have systems that there should never be a need for reinstall. There should be a good support for rolling upgrades, et cetera. Because we're talking about this five level of availability on the service level, we have to have these functions in place. So if we're then talking about a bit what we have actually been doing, since last VLON Tranking, that is really to make the VLON aware VMs to be able to isolate the network all the way to the VMs. We talked about VNF provisioning that is to be able to provide layer three resources to our virtual network functions. Optimized data path that is so that I have a virtual switch that can actually deliver line rate to the virtual machine. High availability. And I think here a lot of the high availability we are focusing on is, for instance, mitigation of failure and reusing the risk of a single port of failure. And that is also included, for instance, that you need to monitor the physical resources. And you need to combine the monitoring of virtual resources and physical resources and make sure that whatever happens, that you have the best possible view of what happened and how you can mitigate it in a very, very quick way. We are also working a lot on the assurance part. Because one thing that a telecom system, even if it's only deployed within one operator, is that a lot of these telecom applications will not have the same SLAs. They will have different SLAs. They may have different views on how to protect the data that is running in the application. That, for instance, means that we actually need to have complete multi-tenant. Multi-tenant in terms of how to handle resources, but also multi-tenant in terms of how to handle SLAs, how to handle trouble ticketing. We should be able to provide a trouble ticket per tenant. So I think here is also an area now where we are kind of putting a lot of emphasis on focusing also the multi-tenancy aspect. Because that will be needed even within one operator. Yeah, so I'm going to talk about one specific thing that Matt's brought up was VLAN Trunking. So this is a good example because it has as many of the facets of the problem space that we're dealing with in OpenStack around NFV. So VLAN Trunking is essential for one particular function, the session border controller gateway piece, which is in the IMS Core and how we manage SIP calls. So today, that piece of software requires direct network access and control over VLANs and Q&Q to be able to create an end-to-end separation for specific calls, which is required for a lot of the phone communications, this L2 separation for that. So that is a good example where today, an NFV or a VNF doesn't really respect the typical IT abstractions of who does what. Really, the OS should be dealing with this particular thing. But because of its past and it's just recently being ported over to this model, it doesn't do that. And it's going to take us a while to be able to do that. And obviously one part of the challenge in this one area is working with the VNF vendors to modify, to look more like traditional cloud. So that's obvious, that's one thing. But we have an interim period where that's not possible. So we, and we want to move quickly to implement solutions around this. We're deploying two enterprise-focused telephony systems today in OpenStack to try to make this work. We don't have time to wait. So we really needed this function in place. And really, it's essentially OpenStack managing OpenVSwitch to be able to expose what OpenVSwitch was already able to do is make a network device on a Linux box, be able to see all of VLANs and have the facility to trunk that. So it wasn't really that hard. And the Ericsson people had identified this as a key thing to make happen early on. So putting up sort of a blueprint of the history of the blueprint to just to show kind of our problems in two dimensions. One is really about what we're doing now is getting people aware of the NFV problem space and helping to promote it. And so Mots and I did a lot of work this way. Hopefully that helps. We've done a lot of work within the community. I spent a lot of time with Red Hat folks trying to convince them to help me to build up an NFV working group that was started thanks to Russell Bryant, very helpful work that he's done to bring together a group of about 40 different people to actually every week on IRC talk about NFV issues. So that's one angle of it. The other is specifically when we look at the timeline for this particular blueprint where the blueprint is saying, okay, we want VLAN trunking. You know, it's been a journey over the last 13 or so months from the point in time when the developer from Ericsson, Erick Mo actually submitted the blueprint to actually making it into the code base and into upstream vanilla. So it's been a pretty long journey. I won't go into the details, but it's indicative of something of a challenge that we have. There's the paradigm shift and the acceptance of sort of having to deal with these legacy systems and sort of an interim solution to solving them. That's one thing. But then also getting at a more fundamental problem with an open stack is just we need more help. We need more reviewers. We need more people familiar with networking and familiar with VNF in particular to help us. And so that's another aspect of it is bringing in and trying to get help from a broader spectrum of vendors and telcos to make this work. So this is typical of actually a lot of things and it's not only a VNF or an AV problem. It's also across all of open stack and something the foundation is trying to work on. So another sort of vector and trying to solve it we've been working on is trying to help the PTLs and augment them as much as possible to make it so that we increase the kind of flow of blueprints and specs to get to manifestation. So that's one part of it. Now, these are some of the examples that I talked about already of how we can help make this change. So pushing on the VNF vendors and working with the community. We felt like in the syndrome time that even more can be done. And this is one reason why we started the OPNFV effort is actually take the best of a standards body like Etsy and the best of open stack and put them together and create something new that actually would focus on NFV and in a practical way do allow us to create sort of a center of gravity, a group of people, some momentum that has some weight to push requirements into not just open stack but obviously open V switch and open daylight and DPTK and open data plane, these other things that are needed to make this work. And then also create sort of a test suite and integration model for putting all of them together and making them work. So this is obviously a broader problem than just open stack. So this is our attempt of even going down the road even further to make this more open stack like. And then hopefully in the next quarter or so we'll see actually certifications and sort of code come out of the OPNFV. Okay, now we are going to a demo and this is always interesting when you're trying to do it live actually here. So what we are doing, demo and this we have been working with the agency for a year or so. This is really, this is a control server of an IMS system called CECF. And this is about how do we actually do monitoring and how do we handle the compute failure that the whole host goes down and demonstrating some of the HF features that we are kind of doing. So take to kind of keep the service level up all the time. And we are using, so to say, the provisioning of the VNF we are using open stack. And we also are using as I said, a cloud manager. And as you can see here, we are defining, so to say, and I didn't talk about that in the beginning there, but of course affinity and anti-affinity is of course a very thing that is very important for this type of application and work loads. So in this case we have, so to say, and maybe the name system controller that is not, so to say, that is the controlling part of the applications and PL that is the traffical part of the application. So PL3, PL4 is handling the diameter signaling, PL5, PL6 is handling the traffic or the SIP signaling. And as you can see the system controllers is defined as one anti-infinity group and the traffic handling part as another one. So these are, this setup is running on standard HP blade servers. We are using as I said, a cloud manager for the orchestration. We are running, so to say, there is now I'm only showing the two VMs for the traffic handling part, the SIP signaling part. We have of course a lot of other, I mean, IMS is not only one application, there is a lot of them, maybe too many sometimes, but you have the subscriber register, you have a lot of other applications that is running on other VMs in the setup. We also, in this case, are using traffic loading tolls. We have roughly around 2,000 SIP subscribers registers, not all of them are of course making calls at the same time. But we are, so to say, establishing new SIP session at the rate of one per second. So these two VMs is handling one instance of this traffic handling part, the SIP signaling part. What is also maybe important to state that there is session replication between these two ones, but that is handled on the application level in this case. So it's not handling on the per VM level, it's handling on the application, so the application actually replicates the sessions all the time, so between these and there is a load sharing between them. So what we will demo here is to bring down one of the hosts where one of these VMs is running. What you will see here is that since the sessions were already replicated from start, so to say, more or less you will only kind of go activate the sessions on the standby side from kind of passive to active. So that means that there is kind of nothing will happen with the sessions that are already established and up and running. We will then create a new VM on a new host and you will also see that soon that that VM will start executing traffic and it will start the load sharing happen to that VM. And this will happen without any service interruption on the actual service. So this is the core control server of an IMS system. So this is what is handling voice, so say in an LTE system. This is a more detailed setup of all the hosts that are involved and the Infinity Group. As you can see, they are distributed. The SICK is the Cloud Infrastructure Controller, that's OpenStack, you have the two and the Infinity Groups for the controlling part of the application and the traffic handling part of the application and then you have the things like Horizon DNSs, HSS is the subscriber register system for IMS. And then there is another one for handling like the tools on the Goys that we are using. So are we ready then, Svettu? Yeah. So here you actually, if you remember the slides I was showing, you have the system controller, you have the four payload nodes so the traffic handling parts of the application. What you see here is the amount of SIP calls that is ongoing and you see there is a load sharing between these two and it's roughly around 130 calls per one. And as I said, all the sessions are replicated between these two application instances and of course not, as I said on application level, not on the VM level. So are we then prepared to kill one of the, so this is running downstairs in a booth. So this is, we couldn't bring it up all the up here. So this is, so I don't know if anyone can run down and check that we are actually rebooting it in real time. So after a while now you will start that it's kind of graceful, it takes it down. Hopefully now you saw that it stopped handling the traffic on that payload node and moves over all the traffic so now all the incoming SIP sessions are going to one of this VM and this application instance. And then that swapped to yellow and that means that it starts to evacuate the VM. So after roughly 45 seconds or something it will notice that the blade is dead or not responding. So it's just hoping that things are working. It's good fun, but I mean, when you're running live. Yeah, so this is actually, as I said, the demo is down in a booth and it's running live down there. So hopefully we'll start seeing some alarms popping up. So now it's evacuated in the VM and you've got an alarm for that one, I think, an SNMP trap. Yeah, and now it's gone completely red. That means that it's taking out, so it's, yep. So quite soon we will see it spinning up. Another VM on the 11, there it comes. Still not handling traffic, but it's up. Yeah, so this is a VM and then it was spinning up the application instance. And after a while it will balance the traffic for both these. And as I said, this is a real IMS system and it's happening in real time. I don't know how long it takes before it's, but I think this is just giving you a bit of examples of what type of additional features, et cetera. Yeah, it's up. Good, worked. Thanks. So this was actually, as I said, I mean performing live and that's why we couldn't really completely control the timing. But during this whole period now, there was no traffic lost, I would say, because the sessions were replicated on application level. Of course, during a while, you had to take the double load on one of the application instances. So of course, during that time, you're of course very sensitive if that one fails as well. That's depending on how many instances you have, how secure you want to be. But this also shows that how can we implement this type of five-nines of availability. And this is of course, a mixture between things that we do on the application level and things that we do on the virtualization management level. And I think that is probably how it's going to be that some things we will actually not do on the virtualization management level because it's maybe too complex. Okay, so that was the demo piece. So we didn't have the complete cloud management solution, et cetera, in place, so this is why we only show in the SNMP, yeah. So, we also have, as I said, we're also showing a number of other things that we're doing and actually showing them as live demos. Another thing that we also have been working a lot with AT&T is what we call the Ericsson Virtual Switch, more or less providing line rate without really the need of an SRIV, so to say, without imposing hardware requirements. OpenStack live upgrade. Maybe I can take them. I will not be the expert in answering all the questions around them. But this is really, as I said, and this is, of course, one thing that I think it's really important for a lot of the network applications that is really line rate to the VM and, as I said, isolation of the network. So here, of course, you have the maintained VLAN tagging, 100% tenant separation, and you have the line rate to the VM. Live upgrade, this is showing, I would say, more of a type of spare wheel upgrade, so that you're moving one instance over to the next generation. And as I said, this is probably not the end solution of upgrade. I think there is needs for much, much more advanced upgrade. But this is shown down there. VNF's orchestration, this is what we're doing with the cloud management. This is really about a lot of the anti-affinity ruling. I mean, how do you create virtual network functions? How do you scale them? How do you decommission them? How do you actually run things that are across multiple data centers? Migration of VM, fault management, performance management, and all this, of course. And the cloud management should be, cloud management by itself should be high availability. Virtual enterprise gateway, that is to say, how do you can actually work with virtual CP? How do you can distribute them? How do you can handle virtual customer premises equipped? Easy VPN, it's about how you can provision VPNs between an open stack part and an open stack-based cloud and an enterprise site. I think this is, of course, something that I think a lot of companies are working with and a lot of companies needs because I think being able to provision VPNs in a dynamic and fast way, so not have to wait for three weeks before the VPN comes up, so I think this is a given need for a lot of operators who understand as well. The last thing that we are demo, which is a company we acquired of Sierra, which is a policy-governed platform as a service, that is a pass layer. And I think what is really, what we're trying to push here is, of course, the governance part of the pass layer, because if you start building this complex system and things start to move, you start to extend your system, it's like, how do you make sure that the characteristics you required actually are executed? And I think we will more and more see that we need a lot of advanced policies that we need a lot of advanced policy functionality in order for making these clouds that we are now doing to perform. So, Toby. Yeah, so I'll just close out with one last slide talking about in the next period of time, the things that we're focused on and where we need help. So IPv6 is in Juneau, a good portion of it is finally in the code base. There's a few gaps still left to resolve that, but obviously for us, in the mobility space, IPv6 is huge. We can't have as many phones as we have, billions of phones out there, or now with M2M and Internet of Things. Just heard this interesting fact. Just yesterday, we released the numbers for last quarter. It was the first time that the number of devices for Internet of Things that we sold was greater than the mobile devices that we have for people. So I think that's just a trend that'll continue to explode and obviously, this is an essential function for making that happen. The policy story that Matt's last left with is essential. Being able to guarantee certain characteristics like packets per second or some level of separation or authorization to do something. Many different realms of rules have to be applied to a system like this to make it deterministic and really do what we've contracted with customers to do. So this area needs a lot of work and I'm very hopeful that we're able to do something within the OpenStack community to do this as well as I'm very bullish about the acquisition of Epsera. I really believe that policy needs to happen a priori within this tools, it can't really be grafted on. The performance, as Matt's talked about with the enhanced or Ericsson vSwitch, any work around this area of increasing the throughput of network through a X86 host is key. So we're doing a lot of work in OpenVSwitch and testing out with Ericsson's vSwitch and then also working with SRRV and Intel trying to find a way to make it so that a host can do what we needed to do. This will be the key kind of boundary expanding work that we do to make this NFV work happen. Obviously, I believe that this isn't gonna work for the Tokos unless we have a diverse set of workloads running on it. From the back end IT systems to the NFVs, they all have to run on this, big data, whatever unused asset we have, we need to make it work but it has to be secure. We have to guarantee reservation and separation. So that's gonna be another area. And then I think the last two things are combined together. We will be deploying 150 or more OpenStack sites, separate distinct geographically different locations. In order to make that happen, we have to make OpenStack easy to deploy, easy to upgrade, easy to manage in a holistic way. So good works happened already with Keystone and with other areas like that where it's obvious, like Glance and such, where you don't wanna be having your images everywhere. You wanna have it deployed once and then it spreads out. Keystone, you don't wanna have authorization done all separate, it has to be in one place. Same thing will apply to the other areas. So these are, I think, six key areas that we'll be focusing on in the next period of time. All right, I think that's it. Any questions from the audience about this area? Feel free to use the mic in the front or the back. Hi. How much are you right now for services like VIMS, VEPC, just replacing physical machines with virtual machines and how much are you re-architecting those services to work better in the cloud? That's a very good question. So I mean, for the three major components of mobility, it's really just a porting of what had been in hardware and moving it over. Now, and I think the first initial deployments will be that kind of legacy. It's gonna take us a while to re-architect those types of systems to the fully cloud scale out model. Though, frankly, they did show up with a lot of this concept already embedded, so it's not so bad. But certainly, when it comes to managing load across many different devices, it had already been pretty good at doing that. When it comes to vertical scaling versus horizontal scaling, that's where work has to happen. So that's where the whole midget cattle come in. Yeah, and as most of the telecom application have been, as I said, have often good horizontal scaling on application level. And most of them are kind of plain Linux application. I think we have had examples of applications that are, for instance, have been relying on Linux kernel fixes. I think that is pretty hard, so to continue. So you more or less, you have to move the... I mean, there's two things you need to kind of virtualize a network function. One is, of course, that it becomes a pure Linux app. The other one, which I think has been a major challenge for us, is really to separate the management piece from management of infrastructure, management of our application. Because a lot of our applications, you have actually been managing the infrastructure for that application through the application. And I think that you need to separate the management. And I think that has been, I think, the major investment for a lot of applications changing the management structure. Yeah, exactly. I mean, we want one more contiguous orchestration across many VNFs and not one specific orchestration for each VNF. So that's a key concept. Yes. In our setups, we use Solometer, but you can talk to this. But we all use the telemetry in other parts, yeah. But you're pointing on a very, I think a challenge is that a telecom system really need a notification mechanism and not a polling mechanism. But I think that is how OpenStack is a good thing to do. All right, any more questions? Oh, yeah. Hi, my name is Salman. So the focus will be on the high availability. I'm wondering, are you looking also towards the mobile protocols to make it more resilient and more better because there are solutions available in that direction that allows you to make more better solution? Yes, I mean, I think in general, I think you can actually achieve a lot of good resilience, having good network protocols, so to say in between. And I think we talked about that I don't think the best way is to take the existing application as is and only move them and try to kind of do whatever we did to create this type of characteristics on the box. I think we need to use a lot of other tools. And for instance, resilience on network level by kind of using mobile protocols or whatever, so to say to achieve resilience. I think that is things that we are looking into, yes. Thank you. Yeah, so that's a very good question. So I mean, the question was about whether or not we see a unified sort of VNF manager or independent for each VNF having its own manager. So I mean, in my view, there will be some common infrastructure that we can reuse across all VNFs. So like autoscaling and templating of how it gets deployed and scaled, I think that's a good example of how we can use heat and its autoscaling mechanism as a sort of common denominator instead of relying on each VNF manager and its experience. Now, I think there's gonna be a balance for a while because the VNF managers that we use today have a lot of inherent knowledge of each VNF and its business logic and such. So it's probably gonna be a mixture. And you see, I think, quite a bit of integration with message queuing and those types of things to make it all work together. So I think there'll be a balance of the two at least at the beginning. Over time, though, where there's commonality you want it to be in one thing. And it's actually defined as a kind of a step two in OpenNFV to look on the VNF manager. But I think that the step one is really to focus on getting a agreed reference working platform on the infrastructure as a service layer as the first step. Exactly. All right, any more questions? I think this is good. Thank you.