 Hi Yeah, it works Hi everyone. Good afternoon My name is Yongsheng Gong. I'm working for 99 Cloud in China Hello, good afternoon Austin So my name is Vish Jaraman. I work for brocade communications So today we are here to talk about an enhanced platform a very deployment of VMs using Tacker so at the end of the presentation. Hopefully would have left you the impression that tacker and Tosca templates is all you need to simplify your lives to deploy VNFs high-performance VNFs so I know the trend is now actually Virtualization and of course trying to actually virtualize network services. So there's a shift away from using Dedicated proprietary hardware for network services and are moving into the Cots platform wherein a similar similar performance that you can actually get from a dedicated proprietary hardware you can actually get in the Cots platform as well and how you can actually use tacker and Tosca templates to specify these requirements and Leverage them to actually deploy your I perform VNFs is going to be the main talk of the session So here what is the agenda that we have? Yongsheng is going to talk briefly about tacker and NFV overview I'm going to talk about some of the features that are available today in a Cots hardware that can give you some very similar performance of that of a physical proprietary network function hardware and This is not just efficient. You also have to actually do some Setup behind the scenes at the BIOS level at the OS hypervisor level and at the VIM level in this case OpenStack There are some setups that needs to be done by an admin before you are a compute node Becomes ready for tacker to deploy VNFs. So that is something that Gongsheng will actually walk through in some detail and Then I'm going to show you how life becomes very easy using Tosca templates and tacker to deploy your performance high-performance requirement VNFs followed by a demo So take it away Yongsheng Introduce the ESTIs and V reference architecture framework We're on the right of the picture. It is the manual part. The manual part that comprises three parts On the top is the MV orchestrator. MV orchestrator is responsible to manage and control MV resources And then is the VNF measures. We have measures to which is to which is used to measure the life cycle of the VNFs and then at the bottom it is the VIM and And on the left it is the external sequence This reference architecture framework also defines some interactions between internal sequence and the manual parts What is tackle? Tackle is tackle is compatible with the MV reference architecture In tackle, there are three models that correspond to the three parts of this At the bottom is the over stack and VI. We can easily resist the over stack environment into the tackle manual environment And on top of it, it is our Tosca template that is to describe our VNFs And and we will introduce the VNFFG in the MV orchestrator to implement the SFC-like features Okay, tackle project is one of the official over stack projects And it has its own core teams, PTL and core members And the tackle project country is not very big. Maybe there are some 30 30,000 NAF codes and we also have Yonich and the function testers that is running in over stack CI sequence Tackle's release is under the over stack release management sequence. So if so, the VTACL release have our tackle packages Tackle's focus is on FV. So MV offers new ways to design and deploy and manage networking services One of the main points of MV is to decouple the networking functions from proprietary hardware appliances And move these network functions to the commercial off the shelf hardware And via the virtualization But if we run the network function Over the virtualization, there are some technologies to consider to improve the virtual network functions performance So let's introduce some concepts to improve the virtual network functions Yeah, there are some features out there in this commercial off the shelf hardware And also there are some hypervisor and OS features that can actually be delivered that can be leveraged To provide near line performance that you can actually get from dedicated hardware So I'll walk through some of them. You might be familiar with some of this But just to give you guys an overview I will actually walk you through some of these things So most of you must have heard about NUMA Some of the things that at least that NOVA has already implemented and tackle leverages From the open stack BIM through NOVA and also from Neutron That you might be familiar with are NUMA topology, CPU pinning, SRIOV and huge pages So these features can actually give that sense of a dedicated resource That you would actually get from a proprietary dedicated physical network function So these are now available in the common x86 architectures That are available commercially off the shelf as commodity hardware So let me actually walk through, give a brief overview of some of these features This is to just remain, if you guys already familiar I mean I want to just give an overview for the benefit of others So you must be aware of that most of the servers today come with multiple physical CPUs With each of them having multiple cores and then the way the memory and the IO devices are placed These devices are actually placed, memory and IO devices are placed close to some of these physical CPUs So which that means they form something called as a node, NUMA node So the more closer to a certain physical CPU and memory and IO devices are The access time for memory or for IO devices it becomes lesser As compared to trying and accessing either memory or IO devices on another physical CPU You have to go through some link, inter-CPU link that introduces some latencies and delays So which is not something that you want in a high-performance NFV workload So that is what NUMA is In general within your having IO devices and your memory closer to a certain physical CPU And you want to actually have your workloads or your memory access and IO access to be placed on those cores That are closer to those IO devices and memory So what some of the papers are published is that it's in the order of about one and a half times If you were to try and access either memory or your IO devices on a NUMA node Which is on a memory or IO device that is closer to a different CPU So if you are actually, you want these devices to be closer to the CPU where you are scheduling the process So that is the key thing And the order is about 1.5 times more if you were to try and access it on a different NUMA node This is just to illustrate that mainly initially we are working on memory So the same thing applies also for IO devices as well So you try to place your IO devices closer to your physical CPU And you have that particular process that accesses that on a core that is closer to the CPU Gives you better performance and reduces your access latency And those are very good for your NFVI performance requirement workloads So there is also a concept of, I think we covered most of this There is also a concept of something called CPU pinning So by default usually either your hypervisor or your OS oversubscribes some of the CPU cores out there So it will keep scheduling and it will keep bouncing some of the processes between this course And this is not something that is desirable for NFVI performance workloads So you want your CPU assignment to be static and you want it dedicated So just like how you have your physical proprietary network functions which has got dedicated resource You will be able to do the same thing using some of this cart's hardware using this concept called CPU pinning Which requires some configurations in the OS and hypervisor layer So another one is huge pages, this is the OS or the hypervisor layer So most of you are probably aware that today's cart's hardware actually come with huge amounts of memory like 512 gig to upto terabyte And usually memory, the OS or the hypervisor chops it at like 4 kb sizes And then it maintains a table mapping all this multiple 4 kb sizes So this actually causes some issue with some cache lookups So there is actually a feature wherein it's a possible add of time that you are able to dedicate some memory pages Like upto like a gig, there are 1 gig pages available nowadays And then you also have 2 MB huge pages available today And most of your NFVI workloads usually have requirements like you know in the 16 gigs or 8 gigs with some CPUs So it is possible that using this huge page feature wherein memory is set aside ahead of time You don't have any of this TLB buffer look aside wherein you actually introduce latency to look up the memory and map them So this can be done away with and can actually lend itself very well to the NFV workloads On the other one, I'm sure you guys are familiar with SR IOV single root IO virtualization Here where nowadays you have NIC devices wherein it can appear to a PCI as multiple, a single port can actually appear as multiple ports to a PCI bus That means you have a concept of physical function and virtual functions That means a single port can actually present multiple virtual functions to the PCI bus And then these can be used as ports and can be attached to a VM So the idea here is that you are not going to be using the hypervisors any network layers which can introduce latency So you can do away with it by actually connecting this virtual function that is presented by this SR IOV capable NICS to the VMs That way the traffic from the VM passes directly to the physical port And that this can actually help with reduction in latency as well as increased throughput So these are some of the concepts I wanted to introduce but before Tacker can use this the admin has to go off and prep the system The admin so that the end user who is actually trying to deploy using Tacker, using the simple task templates There are certain configuration changes that needs to be done at the BIOS level, at the OS hypervisor level and at the VIM level In this case OpenStack to actually realize some of this So Gong is actually going to walk through some of this Yes, and before we can use this, let's use the technology to improve the performance of the VMs We have to set up the sequence, do some settings in BIOS and OpenStack components to prepare the NVI sequence Okay, let's look at the first the NUMA And to enable the NUMA first we have to get into the BIOS sequence This is my BIOS sequence, maybe different BIOS sequence have different configuration items In my BIOS sequence it is not interleaving to set the not interleaving into disabled We will enable the NUMA to project into the sequence After rebooting the sequence we can log into the NUMA's operating sequence By running the NUMA CTL command We can show that the operating sequence is viewing the NUMA topologies And then it comes to the hybrid threading There is also one configuration items in most of the sequence Let's get into the BIOS sequence and find the configuration items In my sequence this is a logical processor We can enable it or disable it to enable and disable the hybrid threadings Again we can log into the NUMA's operating sequence to verify that Whether the hybrid threading is enabled or disabled For SROV we also need to get into the BIOS sequence to locate our Ethernet adapter Under the Ethernet adapter we will find one configuration item such as virtualization mode We just set it into the SROV mode After we enable it in the BIOS sequence, in the operating sequence level We also need to do more jobs The first one is to pass the color one and one more parameters such as intro, IOMMU, set its own We also need to give the network drivers more parameters when it is noted by the color sequence Again we can use the NINIS commander LSPCI to check whether there are some virtual functions that will be enabled When we talk about CPUP and we need to enable it in the color command line That means that the NINIS color schedule will not schedule processing onto this physical course After that the NINIS operating network functions can run on these CPUs To enable huge pages we also need to pass some parameters to the NINIS color After that we can log into the NINIS color sequence to check what type of huge pages are set And how many pages are defined With the NINIS BIOS and the NINIS operating sequence set, we need to set up the OpenStack components First it is a normal schedule service We need to add two more filters into the normal schedule The first one is a normal topology filter The normal topology filter will allow the normal schedule to know the topology information from the normal computers And the PCI pass-through filters will allow the normal schedule to schedule requests based on PCI devices information on the normal computer On the normal computer systems, if we want to enable huge pages, we need a new network The new versions should be native to 1.2.16 We also need to modify the normal.com file to use NINIS NUMA or VPU settings The first parameter is the VCPO PIN set VCPO PIN set should be equal to the color parameters that we have set before And we also should net the CPU allocation ratio to 1 and to avoid over-committed CPU allocation And the third one is PCI pass-through wireless That is for SRV allocations for a particular neutral network Let's come to the neutral part To enable the SRV settings, we will add one more McKennon driver into the neutral server That is SRV NIC switch McKennon drivers The NIC McRae driver will set the SRV information onto the neutral parts We have to know the PCI vendor ID for SRV card on our computer node After that, on the control node for the neutral server ML2 plugin, we have to tell the neutral servers which PCI vendors are supported And in Mitaka, we have to enable the SRV agent On the computer node that is running SRV agent, we have to set SRV NIC on this normal computer The physical network cards are used So with this, we have prepared the sequence and the over stack I will walk you through how TACR and TASCA make it very trivial Why shouldn't you be using TACR and TASCA? Once you watch this, you will feel that this makes my life easy You have all these complex features out there The next thing is you want to be able to use it But you don't want it to make it as complex as the setup you have You want to be able to express it in a way These requirements make it easy for an user to be able to specify and have TACR deployed for you So I will walk you through how simple or trivial some of these templates are in expressing this As you can see, one of the complex examples I picked is Not many would do this, but you probably want to have your NFV Wherein you want to say that I want to actually have 6 vCPUs And I want to actually have 6 gigs of memory but 2 gigs on NUMA Node 0 and 4 gigs on NUMA Node 1 And you want this kind of requirement to be met So with TASCA template that we have introduced a new property The TACR team has actually been working very closely with the TASCA body to introduce some of this With TACR being the experimental phase where we try this To make push-through standards And we have introduced this concept called NFV compute Where you can provide properties And the situation has described Wherein you want to actually have a non-uniform resource separation between the nodes can actually be specified here You give it a specify a node and give it an ID You say how many CPUs, I mean what CPUs you want placed on that node And what memory allocation you want Do something very similar on the other node You have specified this complex thing In the TASCA template, TACR will take this VNFT It gets uploaded and our VNFM will take care of actually meeting these requirements for you So it's as trivial as this So the next one for SRIOV Of course when you prepare your interlay you will have actually specified your SRIOV network, your compute nodes The only thing you need to say is I want an SRIOV type connection point And the virtual link is actually of this particular network name Which is a SRIOV network And everything is done magically for you by TACR And using NOVA and Neutron Orchestrating with that will get this for you done So isn't it trivial? So NFE is no longer complex deploying You want to do dedicated CPUs It's trivial again You are saying your CPU allocation you want it definitely to be dedicated And you don't need to go off create flavors You don't need to go create post aggregates or any of these things on NOVA Or create any flavors like I said So you specify we will take care of parsing, TACR will take care of this requirement Make sure the interlay is set and then your dedicated CPU is there Your NFE Same thing for huge pages So here in this particular case I'm using a single example of saying we want a large Huge page you can actually say specify 2MB, 1GB, any other Dedicated any other huge page size that is supported It can be specified here If you didn't do the configuration so it will actually very similar to NOVA It will find no validos phone You'll get an error back saying no validos phone So the underlay has to be prepped like you walked through earlier So admin has to set that underlay You should still be able to deploy Oh, okay The question was what if some of the steps that Gong walked through in preparing underlay For your like huge pages, SRIOV or for new multi-plug if it is not set What would happen? So in this particular case if you are requesting a huge page Huge page size of large and there's not a compute host That has a huge page size it will say no validos phone But if you don't want those huge pages or any of these performance parameters You can still go off and deploy a regular VNF without any of the special requirements It would still go find a host and replace it there So with that some of the templates that I walked through I have actually got a one minute demo for each of them to show How you specify the template How you use the VNF manager to actually deploy the template And actually have it realized through horizon And we'll also go look at how the KVM or the WERSH utility We do the dump xml to show that what you requested We got it on the compute node So let me share this video So what you'll be seeing in this video is Showing a VNFD template, Tasker template Where we are specifying we want a dedicated CPU In this case it's a 2vcpu And once it gets deployed I'm using edge top It's a utility that tells you on your compute host how the processors are actually running So I'm going to show that I have this template deployed And it does go and take two CPUs and it's dedicated So that was the main thing I'm going to be showing here So here is where you see those are the two where it gets deployed 9.11 So there's no workload running Now I'm actually going to go off and In this year I'm showing the WERSH dump xml to show that The two CPUs that LibWord sees they are pinned And next I'm actually going to go off and run a workload stress test So that those CPUs are actually running at full capacity So if you see edge top that 9.11 CPUs numbers they are actually loaded So that was the CPU pinning Next I'm going to walk through huge pages So when you actually log into a compute node like Kong had shown earlier Excuse sorry about this So you can actually do a grep on the proc meminfo And you will see how many huge pages are used In this case they were 24 and they were not used Now I'm actually showing how this can be specified in the template In every template and once you deploy it We will actually go back to the compute node Look at the proc meminfo and you will see that it has actually used up some huge pages So you will see that it's now 22 So two huge pages because it's a two gig RAM has been used up Now we are going to go into verse and also see that The huge pages are actually being associated with CPUs The next one I'm going to do is a new multipology demo The one that was the complex one Look at that So this is the complex new multipology that we specified And we are going to actually see again And once it gets created in lib I do a verse dump XML of that instance And it will actually deliver what you requested So the key is in showing that this task templates Authoring them and specifying this performance requirements is very trivial And have this an app tracker interpret this And be able to meet the performance requirement So there you can actually see the new multipology Where the CPU sets and the memory are as specified The last one is the SRIOV This is where we show the So we will see that in this particular case the PCI device The physical link that appears now with multiple virtual functions To the OS One of the virtual function actually is consumed By as part of creating this VNF And when we do a neutron port show On that particular port you will see the associated physical The PCI device info being used So again the emphasis on all these demos is that You can use task templates Some of these requirements can be specified In a simple easy to understand manner And have tacker work with the open stack frame To go off and realize this So here I'm showing the IP address That I'm actually going and correlating And trying to figure out the corresponding port You will see that it's got the PCI device information Which I will be highlighted here That concludes the demo and the presentation So open to questions If you can get to the mic that way it gets recorded So that definitely looked like a bit of setup I mean are there efforts to try to simplify or automate that Or is it something that let's say the BIOS settings for example Is it something you can actually automate as part dynamically Of setting it up before you launch something So is that something that tacker does? Is it possible? Yes there are actually I know There are companies like HP and Dell who actually sell servers They have some utilities out there They have their own out of hand utilities So it's not a standard it's a proprietary It's something I think they have The utilities wherein you can prep the underlay So how does I mean obviously you showed some commands Where you can like introspect to see things are set properly right? But in the work for deploying a VNF Do you need to do that in order to know Actually that it's not going to fail Or do you like mark your host resource With these attributes based on some prior test Or how is that prearranged to know that as Dan said That this will actually work So usually the admin or the operator Will have a sense of what kind of workloads Are going to be deployed in this open stack instance And he's going to go to the prep and marking Like you said identifying which host Either you can use things like host aggregates That is provided by NOVA to mark them Or availability zones Or they're going to have it there And VTacker will actually go create some of those It will take away some of the complexity And identify the hosts that has these features In conjunction with NOVA and will deploy Okay so we would have to segregate and prearrange The configuration of the bare metal host With those features Yeah you can have a mix of it You can have a mix of hosts that are going to place Some high-performance VNFs with ordinary hosts It can be mixed up As long as on the compute nodes You do some of the underlay preparation That Gong actually went through So I have two questions One is with Tacker Can you apply these changes Or the VNF enable these enablements for VNF Across a pool of compute nodes Can you automate that process with Tacker Which exact process are you talking about? So setting up the parameters The grub modifications The reboots Make sure the parameters have taken over These are not Tacker specific functions Okay So just to show an end-to-end flow We want to share the solution What all has to be done And now Tacker can simplify So Tacker itself will not actually do any of those things for you I see So we might actually as part of our documentation From user documentation we will have this And there are also other documentations out there From MOVA and stuff So Tacker itself will not do this Tacker what it does is It is the one that actually goes off And orchestrates and figures out The mapping The VNF and places it The second question is SRIOV And link aggregation protocols Bonding Have you had any experience with that Do they work together? I don't have that experience You have any? Okay, thank you I think we have one more minute for a question Can I ask one question? Sure I want to know your view Because with SRIOV there are certain limitations Like live migration doesn't work Okay So how do you think This will be solved in future release? I know that SRIOV At least from what I have read SRIOV as compared to PCI pass through PCI pass through doesn't allow for live migration But SRIOV has that capability But it might be a vendor specific thing Probably could be SRIOV from what I have read Or researched Seems to have that You can add to it As the migration capability SRIOV it has a capability Intel has also done Open stack So that is something If that happens on a nova level I think we should not have any issues from Tacker to be able to leverage that Second question One thing is of course these parameters Will help you for a performance game But what's your approach for benchmarking Or testing these capabilities I don't think rally and tempest is good enough If I am using Numa And I am getting better CPU performance And so I don't know On your real experience What do you guys use In your labs So I myself have not been part of any of this Performance testing But I know do know folks who have done that And I have actually proved either on VMware or on other OSS They have done this I need to go find I mean and see if I can share some of those links But in my case I have not done those Any of those Reports Where on standard systems These have been tested and been published Because it's really complex Performance on Numa Node 0 Is different from performance on Numa Node 1 Okay thanks Fair question Mine is more a question on future direction So if you look at application templating We have heat and hot templates Now for VNFs and Tacker We have tosco based templates Is there any effort across project To harmonize on one formatting Or more ease on deploying VNFs And applications the same Instead of learning two different Templating languages So from what I understand I might not, I mean one of my Colleagues Shrita might be able to answer this better But TASKA seems to be the way For both application And also for workloads Including VNFs That seems to be the standard It seems like So you are saying heat will change Today the WIM that we are using is OpenStack Let's say you would actually Have a heterogeneous WIM OpenStack You have just standalone KVMs Or VMWare I think TASKA is probably still The one standard thing that could help So I think we come to the end of our questions So that's what actually I feel is the standard Using TASKA and heat is very specific To OpenStack at this point of time That's the thing If we have any more questions We can actually have a discussion offline Thank you Thank you so much