 So it's time to start. Hello and welcome. I suppose this is the last presentation slot in the main summit side. I'm pleased to see that you still had energy and interest to come and see our presentation, which is open stack with real time applications. My name is Juha Ravainen and I'm working in Nokia in mobile networks architecture and technology unit. Okay. My name is Tapio Talangri and I work in Juha's team in the mobile networks and I'm also the chairman of the technical steering committee in the OPNFE project. So I have a few advertisements for OPNFE also in my part of the presentation. Good. So today's agenda. First we have a short intro, then we talk from radio cloud requirements and last part of our presentation is from real time optimization, which we have divided into three parts. Vim, NFVI and VNF. So what these abbreviations mean? Vim is more or less open stack. NFVI is the actual hardware and the visualization layer and VNF's are the real applications. So what we are doing in architecture and technology? In our portfolio there are all kind of fancy things. Together with Tapio we are working with, among other things, open source, like OPNFE, open stack and some other projects, cloud in core and radio networks, data centers and so on and so on. At the beginning of our presentation we would like to show you a short two minutes video, which gives an overview from our presentation. Work so well that you don't even need to think about them. They're not designed to be noticed. They don't have to be seen. They're just made to work or even to work wonders, knowledge that enables a network that acts intuitively. A scalable network designed to grow with people, offering unlimited capacity with less effort than ever before. Designed not just to meet their needs, but to anticipate them. A network that is made to adapt to people, communities and the future. Whatever big and wondrous things it may bring, the biggest thing you'll ever see is the biggest thing you'll never see. Meeting the network needs of today, tomorrow and the future. Nokia AirSkill Radio Access The mobile industry, as we know, is going through a radical transformation. Just think about smart cities, internet of things, connected cars, e-health. The list just goes on and on. As the future becomes even more unpredictable, the key requirements for mobile networks are agility and adaptability. Adaptability is most effective if it takes place automatically in the background. From network elements to mobile phone applications, we often take technology for granted, without realizing what really happens behind the scenes. Sometimes the most powerful things are not designed to be noticed, like you saw from the previous video. Current activities are concentrating to clarify core network functions, but the next move is to support real-time applications. And to get real-time applications clarified, support is needed from hardware, from cloud stack and from the applications. And we will cover all these parts in this presentation. Telko and IT worlds are merging. Scalability, flexibility, agility are IT cloud characteristics. But at the same time, we need also to support telko key requirements. Cloud needs to be proactive, it needs to be Telko VNF aware, cloud needs to be fast and it needs to be traffic and cost optimized. Next part is from radio cloud requirements. 5G is the new generation of radio systems and network architecture delivering extreme broadband and ultra robust low latency connectivity and massive networking for the internet of things to enable the program of the world, which will transform our individual lives, economy and society. The industry has adopted view that 5G will be about people and things as per the following three use cases. Massive broadband that delivers gigabytes of bandwidth on demand. Critical machine type communication that allows for the immediate synchronous high hand feedback that enables remote control over robots. And massive machine type communication that connects billions of sensors and machines. And flexibility and reliability are the key design principles in 5G. And here are some numbers which visualize how short latencies we are talking about. On the left-hand side, the green bars indicates some human reflection times to get better understanding from these latencies. So 120 milliseconds is the time what it takes when sprinters like Usain Bolt starts running. And if it lasts than 100 milliseconds, then it's a full start. This is comparable to latency. What is acceptable, for example, in cloud gaming. 7 milliseconds is the time when you blink your eye, which is pretty fast. But even that's not enough to 4G virtualized run or 5G. And the latencies are coming from networking and processing times. On the right-hand side, you can see that how long it takes in optical fiber when the data goes 100 kilometers. So the round trip time is about one millisecond. And the round trip time is from A to B and back to A. So like already said, 5G brings latency requirements to an 11. In 3G, end-to-end latency is about 20 milliseconds. In 4G, it's about 10 milliseconds. And in 5G, we are talking about one millisecond or even less. So this means new radio, and also new architecture is required to full low 5G latency requirements. And one other aspect is amount of mobile data. You can see mobile data usage growth in the past years. And the growth has been pretty massive. For example, in Finland, where we are coming from, last year it was about 5 gigabytes per subscriber per month. I think you have all the information here. I just saw that mobile subscribers have finished mobile operating. They were using 13 gigabytes per month in August. So that's pretty big number of cat videos people have been watching with their mobile phones. So this is until 2015 and this 13 gigabytes. It's about two or three times more than last year. So the massive growth is just continuing. All right, so then I hand over to Tapio, who will talk about real-time optimizations. Okay, so now when you think about this open stack, you may think that, okay, open stack is not in the data path. It doesn't have much to do with meeting the real-time requirements. But however, where open stack has a role, it's in sort of deciding where the VMs are running and sort of enabling some of these optimizations in the data path. So here's a diagram that I'm going to refer to. This is the Etsy VNF working group, sort of Etsy reference diagram. The virtualized infrastructure manager, which of course in our case is the open stack, that's on the right-hand side, it's part of the management. And what we have on the left-hand side is this NFVI, the network function virtualization infrastructure, which consists of the hypervisor, the virtual switching and the hardware. And we're going to talk about all of those, because all of those have an impact on the VNF. And then finally, we come to this VNF and how the VNF can be optimized to improve the performance. So I'll start with, so talk about the VNF and how the open stack and how the open stack is enabling the real-time optimization. So I have a picture here of the open compute platform, which is a great open source hardware platform, and I'm actually going to talk more about that. So instead of plates or servers, it has this thing called sleds, that you can slide in and slide out. And on the right-hand side, there's a picture from a ball, which instead of gives you an idea of what the whole thing looks like, there's two CPU sockets, and then there's memories attached to the CPU sockets, and you have a PCI slot in the bottom, and then in that picture you see that you have a riser, and then you plug in the NIC cards to the riser. And here's the schematic about the same thing. You see there's this sort of CPU sockets, memories attached to it, there's a QPI connection between the sockets, and then exactly one of the sockets is connected to the PCI bus, and that way also to the network interface card. So when the packets come in, they go to one socket. And that's actually important. But the first thing we look at is this placing, sort of being aware of this topology. So there's two sockets. It makes a difference whether your application is running all in one socket, all in another socket, or if you have sort of a big application which cannot run on one socket only, then you want to have an idea of how it's going to be placed on this hardware. So in OpenStack these days you have this capability to define the flavor. In the flavor you can define how many NUMA nodes, in other words how many CPU sockets, you want the application to see. You can tell how many CPUs you have on each of those sockets, and then you can also allocate how much memory you have on each of those sockets. So that's one optimization that can be quite useful and actually is used in the solution that we talk about. Another thing is this CPU policy and the CPU thread policy. So the VNF that we're talking about, that's not the only thing running on the same CPU so we can define whether we want to sort of dedicate the CPU cores to our application or whether it's okay that we have other applications sharing the same CPU. And the same thing goes, so that's about the CPU policy. It can be dedicated or not. And then similar thing with the CPU threads. So inside of the CPU we have the cores, inside of the core you have two hyper threads. The two hyper threads are actually sharing same resources in the Intel architecture. And then with the CPU thread policy you can define whether you want to have just a single, okay, if you want to have your hyper thread to be the only hyper thread which is running on that core. And that's one way of sort of guaranteeing latencies because then you know that it has this sort of dedicated resource for itself. And then finally the final piece in this placement thing is this PCI pass-through which is an optimization. You may have heard about this SRROV term. It's the SRROV and PCI pass-through. Roughly the same thing. And when you're using PCI pass-through or SRROV then you're making a PCI device directly accessible to a virtual machine. And as I said, when a packet comes in it goes over the PCI bus to exactly one socket. And then if your VM is running in the different socket then it's obvious that it's not going to be as fast as if the VM is running in the right socket where the packet first arrives. So what OpenStack does is actually if you have a PCI pass-through Opestack will figure out which socket your VM should be running. So you don't have to actually worry about that. You just have to define the PCI pass-through. Then a little bit related thing which is not so much about real time. It's about the fault management. It's sort of something that we have been working on which is very close to what we do. It's this thing where we want to monitor the host, the server. And in the beginning of time a few years ago it was so that there was sort of no monitoring, no active notification to the NOVA DB. If a server didn't respond, then the NOVA just marked in the database that okay, this doesn't seem to be responding. I'm not going to place any VMs on this server. And that was it. But that's maybe not ideal and we can actually good feedback from the operator community about that. What we have now built in the OPNFE and also done in OpenStack is there's this mechanism where you can have sort of an inspector on the server which will monitor all the time the health of the server. You will notice immediately or really fast if the server has failed. There's the vitrasch OpenStack component which will notice it through the inspector and then the vitrasch can mark that this host is not responding and there's a new API implemented to sort of tell NOVA that now stop using this, it's down, it's not working. And then with vitrasch you can also have a notification going to the VNF manager. So if you have some VM in your application which is managing the rest of the VM, that's the VNF manager. The VNF manager can sort of do some recovery actions, for example, before it's like sometimes you can do things like failovers and things like that. And our goal is to make that really fast. We talk about less than one second from the time that the host is failing to the time that you get these notifications going around. Then of course it might take a little bit longer to do some recovery actions if you have to boot a VM or do some application level stuff. Okay, so that was the sort of OpenStack part. Next time talk about a little bit about server hardware. It's very interesting about the hypervisor improvements and then talk about a little bit going to the future with the virtual switch off load and things that we see coming. So now in this Etsy diagram we are talking about the left hand side, Linux, OpenVswitch, DPDK and also hardware. I'm going to start with the hardware. So one direction where we are going is definitely, it's always faster networking. Now we're going on this path of like 25 gigs, 50 gigs, 100 gigs. We have in our system that we have hardware acceleration for crypto, IPsec off loading. We also have hardware acceleration for packet switching and I'm going to talk about that a little bit more. And then there's all kind of little things which allows me to say that it's still optimized hardware. It just have to do with the bios settings and how the bios is optimized and how it's verified and tested and also about like how many nicks you have and what kind of network connectivity you have and things like that. I mentioned this open compute project already earlier. It's very exciting, you probably heard about, I mean it's a project that was started by Facebook and then they wanted other companies to join it and the idea is to make the hardware open and so that you would have like an ecosystem around that area. Nokia is also participating and contributing to open compute project. We have specs like for example 48 volts specification and earthquake proofness of the hardware infrastructure, things like that which are typically important for telcos. Many telcos have central offices which have 48 volts. So the benefits of the open compute project you have more servers per square foot or score meter. It improves the energy efficiency. It's easier maintenance. So with this rack servers you have to sort of unscrew it, slide it out, it's very heavy. You have to put it on the floor, open the screws, replace the components and then lift it back. It takes minutes to do it like that. With the open computer just unconnect the plugs out it goes. So all of this is giving you the lower cost of ownership. Now of course after telling so much about this open compute project, I'm sure you want to see it in practice. So I went to our lab and we actually have the latest Nokia branded OCP hardware and I took a picture. I hope the picture worked okay. Oh damn, there's some. Okay, sorry, you can see the hardware. There's some idiot standing in front of the camera. But anyway, you see the sleds here and the blue colors and the connections and so forth. Okay, so that was about the hardware. Then I'm going to talk about the hypervisor improvements. And now I took some slides from... This is a slides from actually OP NFV summit where I gave a similar presentation about an OP NFV project which is looking at improving the real-time performance of KVM. And what I show here is this kind of round-trip latencies. I mean, which is sort of a nice way of measuring the performance of the hypervisor. So you have a traffic generator, you're sending packets into one interface. You have a DPDK-based forwarder and then that's sort of forwarding and sending the packets back on the other side and then you take the measurements. And you can see that the average latency has improved from 290 microseconds to 20 microseconds, which is pretty impressive. And also the maximum latency has also gone down to about like 100 microseconds. These are measurements that I think we actually did in our labs. And then here's another picture where it's just to give you an idea that there's a big difference in sort of... There is a difference between like real-time optimized KVM and general KVM. And especially the difference is with the small packets, small packet sizes and the latency for the small packet sizes. That's the most critical thing for us. Then I promised to say something about the future, some of the things that are happening. So one thing that we have is this software... Okay, there's this OVS and then there's the currently what we have is quite commonly is the software base, software switch, sorry, V-switch with DP-DK acceleration. And here's some kind of a rough schematic of what it looks like. The packet comes in through the nick and then it goes to a queue. Then there's this V-switch, which actually has the DP-DK Polmo driver. It takes that, then there's the software virtual switch which is doing the switching and then there's the virtual port driver which is sort of implementing the virtual nick interface towards the virtual machine. And then of course on the right hand side you have the OpenStack Neutron plug-in which is controlling the virtual switch. One thing that we're having is this INIC intelligent nick where you actually take some of the functionality away from the software switch which is running in the Linux kernel and you're actually doing that in the network interface card. So you have this model where... I mean anyway in this network interface card you have this embedded switch you have been able to do things like putting packets with different V-land tags to different queues, things like that. So this is sort of extending that idea that I would be able to sort of do some of this flow processing in the virtual switch inside of the hardware which will of course make a big difference. And as I said that you're moving beyond 10 gigabits, you're moving to 25, 50, 100 gigabits, something like that. So this kind of technology has become very interesting. Then the logical conclusion of that is going to be that you have this whole virtual switch inside of the NIC card and that's also quite an exciting idea that you actually have an open flow interface to the NIC and then you can think about NIC like... Okay, you manage the NIC like... Okay, like V-switch, it has an open flow command and then there's some kind of local user engine which is for one of the commands and you still have the OpenStack Neutron plug-in controlling the whole thing. Finally, you have this crypto afloat as sort of the hardware features and this is something that you can actually get already but it's a little bit related to this. So here you have the packet comes in from the NIC, then it... Oh, it should say E-switch. It goes to the VM and then the VM can use a crypto axis, sort of cryptic accelerator, some kind of hardware which is able to do certain kind of computations very quickly and sort of offload those calculations to the hardware device and that's sort of visible as a PCI device to the virtual machine and now from this picture as you can guess, if you have big packets, there's a big benefit in offloading the calculation because it can take a long time to do the crypto algorithms. If you have a small packet, the benefit is not that good because you have an extra step here, you have to go over to the PCI bus to the device and then come back and that's add some latency. So there's a trade-off, you have to be careful about this. One thing I just wanted to mention is that there's a lot of interesting stuff going on, one of them is the yardstick project and I just wanted to sort of show a picture of some of the testing that you can sort of very easily do in the yardstick. The idea is that you like all kind of tests built in, you can just start it and then there's this graffana interface that allows you to view it. What I have here is I was running this Etsy test case 002 which is actually just sending a ping message with different packet sizes between two virtual machines and the scale on the left hand side is I think it's millisecond. This is not a great result but this example is not that great in that sense but it's working, it's less than a millisecond latency in here but there's a lot of different test cases built into yardstick and that can be easily taken into use if you want to run your own tests and have a nice graphical user interface to it. Okay, thanks. Okay, thanks Tapsan. So, Tappi already covered this WIM and NFVI layers and now I'm going to talk about those VNFs which are the real applications. So, this is what we have and continue to have. You can see some 2G, 3G, 4G and some other existing techniques and current mobile networks are like this. It might look like a mess and maybe it is a mess but anyway it's working. So, this is the current situation and this is where we are going. On the left-hand side you can see the physical antenna sites and there is antenna radio frequencies, radios low layers. In between we have transport and then on the right-hand side we have cloud and in the cloud in this case there are these higher radio layers, mobile edge computing, distributed core and so on. And this looks pretty much simpler. And how this non-real-time real-time partition goes in cloudified run. So, the non-real-time part are in the cloud and the real-time parts, meaning the low layers, are in the physical sets. There are a lot of cryptical abbreviations in this picture but let's not go into those details. But anyway, so the main idea here is that the split is has been done between real-time and non-real-time layers. And here is one example from Helsinki City Centre where we are coming from and Helsinki is definitely not one of the biggest cities in the world but even from this picture you can see that in one square kilometer there can be tens of base stations. And this kind of network can carry about 0.5 gigabytes traffic per square kilometer. So this example is from 3G but in 4G density is naturally much higher and there we are talking about different kind of numbers but this might give you some ideas of why radio is also beneficial to put into the cloud. And the steps which we have taken and will be taken in evolution towards cloud. So the physical parts like remote radio heads, part of base stations and access points are in the physical sites in the left-hand side. Then we have distributed clusters and centralized clusters and there I have also some examples from these VNFs which have been cloudified. So E0B is base station. Then we have base station controllers, radio network controllers, Wi-Fi controllers, mobile edge computing, evolved packet core network elements and so on. And because our presentation was mainly focused to radio site, in my last site you can see the radio VNFs we have already cloudified in Nokia. So base station, radio network controller and Wi-Fi controllers. So they are already available. So thank you. Any questions? Total silence. Yeah, so question is do we have a hardware portfolio for the cloud? Yes. Yeah, it's possible to order it. Okay, there was a question about how does this crypto acceleration work and actually I'm happy to be very detailed about this because I just did it in the summer. I was doing testing IPsec performance. I set up IPsec tunnel between two virtual machines and two physical computers. And in the end when I got it to work it was very easy because I just took the latest Fedora. It has all the drivers built in. Okay, the cloud was of course built so that I have this. I mean it was using SRF with PCI passthril. So from the cloud point of view I could see that there was this hardware device. I assigned the hardware device to the virtual machine, the Fedora virtual machine when I booted it. It recognized the PCI device. It loaded the right drivers. And then when I started this strong one, I was using strong one to set up the connection. It loaded the strongest one and the strongest one and the Linux kernel and everything. It was able to just use it. It was offloading everything. And I was able to look at the counters on the hardware side. That just sort of verified that it really was using it. And the performance for big package was really good so you could see that it was doing something. So that's basically, I mean it's not, yeah. Short answer to your question, does it work the same way as with physical one? Yes it does. It's just with SRV and it's a virtual function instead of a physical device that it works. Yes. I think you guys have been heading this demo work. Lessons learned from radio cloud or lessons learned. Okay, it was not easy. And you need to think about that. How do you do it? Like I mentioned in one of those slides to show this real time and unreal time and the split there. So you need to take into account a lot of things because these delays are coming, for example, from the fiber and it depends what kind of backbone you have. So the one solution doesn't fit to everybody. And you need to do a lot of work on the application side because you have a monolithic application in the beginning and then you're sort of slicing it up into both sides. But I think from an open stack point of view it was not that painful that, you know, it was more like the application. Maybe the tuning and optimization. Problems were in the other components. Yeah. There was the real time kernel. Yeah, it was part of the real time KVM. It comes with that. These numbers I showed from the real time KVM project. It was also using a real time kernel. And then it was using real time KVM. Yeah. So yes, it's part of the real time KVM. It's the OPNFV. If you go to the OPNFV website, it's the KVM for NFV project has built this. The sort of, okay, they have built, there's a kernel and everything that they have done and it's part of this OPNFV distribution. And I don't remember which installer it has. Thank you, virtual machines that launch up and then they will install the rest of the system and that's how you get the real time kernel and the real time KVM if you want to test it. It should work quite nicely. Okay, thanks. I shouldn't talk. I lost my voice. Packet, I think we put mainly what we done. I mean, we mainly looking at packet latencies because that's like the interesting thing. I think it's more like the interrupt latencies is sort of an enabler for the packet latencies. So we also are measuring interrupt latencies and also on things at the lower level. But that's more like making sure that, you know, you want to figure out where the latencies come from, whether it's the networking layer and so forth. So for example, I mentioned this optimized hardware. So first thing they always tell you when you do real time applications is to take real time hardware. So I was doing that kind of testing to sort of measure the interrupt latencies to verify that I'm not getting any extra latencies from the hardware because with some hardware you do get long latencies like millisecond latencies and then it's good to know where they come from. I'm sorry, I didn't understand the code. No, I didn't get it. Yes? Yeah. Okay, good question also. Do you have a packet that's going long way, which one is the slow one that can really tell immediately because always when you do this testing you sort of, you only can test end to end. You cannot tell where the thing is that... Yeah, sorry, I can't really... I mean, it's obvious, I mean with things like SRV, reason to use SRV is that you can bypass some things like the V-switch for example. You can go directly to that. That would be one way of sort of comparing the V-switch overheads. Okay, thanks. I think that we have used our time and... Thank you. Kiitos.