 Hello, everybody. So good day to everybody. We'll be talking about Cyborg today. Cyborg is an open stack project to handle accelerators. So we'll be looking at the general data center trends which require the need for accelerators in the first place. And we'll be looking at some typical use cases for having accelerators. We'll be having an extra focus on FPGAs because they're typically a bit more complex and have more requirements compared to GPUs and other devices. So we'll be taking an additional look at that. And then we'll look at how open stack Cyborg architecture actually realizes those use cases and what we plan to do in this time release. And feel free to stop me at any time if I'm going too fast or too slow. All right. So let's first look at why we are even looking at accelerators in the first place. So if you look at the traditional data center workloads, if you're running a database or a mail server or something, your traditional Xeon type of server processors are good fit for that. In fact, they're optimized for that. That's what we've been using over the decades. But there are new and emerging workloads which are very compute-intensive. You execute on a general purpose processor. So for example, if I take machine learning or video transcoding or any other newer ones, they tend to consume a lot of cycles. So we can optimize Xeon by implementing new instruction sets like AVX, et cetera. But beyond the point, it's more performance and power efficient to offload this computation to other devices like GPUs, or Quick Assist, or FPGAs, and so on. So there is an increasing trend towards using accelerators in data centers. You might have heard about Microsoft using FPGAs. Of course, GPUs are very, very common. When you say accelerators, people actually think of GPUs, of course, normally. You might have heard of TPU from Google, the TensorFlow processing unit. Amazon has something called Nitro, which is to offload the host-level processing of things like network packet processing to accelerators. So talking of FPGAs, there seem to be increasing in the news. A lot of people are deploying them. Like Microsoft Azure, for example, uses that. And we are seeing other public clouds, especially in Asia, deploying them recently. So why is that coming into work these days? So FPGAs are pretty old technology. It's been there for many decades now, but they're kind of getting a new wind, a second wind these days. The reason is this. If you want to offload a computation, your broad choice in terms of hardware is either an ASIC or a programmable device like an FPGA. If you compare the two, ASICs have a higher initial cost because you've got to do a design. You've got to prepare your mask and all that stuff. You've got to spend additional time on design effort up front. It takes more effort to get that first unit off the block. But once you do that, every additional ASIC has a relatively small incremental unit cost. Whereas if you look at FPGAs, the initial startup is not that much. It's buy an FPG off the market and start working with it. And after that, beyond a certain point, you'll find that the unit cost is actually a little higher. In terms of the incremental margin cost. So that's a kind of a flip over at a threshold point where ASICs become more economical. But over time, the initial cost of an ASIC is going up. So the threshold for the blue line is going up. That's a starting point. And the incremental cost for FPG is not as high as it used to be. So the threshold point is shifting to the right. So the net result is, where we consider the FPGAs to be a suitable fit for prototypes in the past, not as fit to be used as a regular offload device by itself. The other major trend we are seeing is that agilirator volumes are getting fragmented. So let's say you are doing HPC. Let's say you want to implement some genomics algorithm like Smith-Waterman. It's very unlikely that you'll find enough market to create an ASIC just for that or an ASSP just for that. It's much more economical to use the FPGA for that. So the fragmentation of the acceleration market is also a factor to note. And the third big factor is that FPGA is very versatile. They actually come with various IO transceivers built into them. So you've got PCI and Ethernet, et cetera, built into that. So you can actually implement networking workloads. And you can do a lot of things which other offload devices like GPUs cannot do. So this is why we're seeing FPGA is increasingly emerging. But the versatility of FPGA also means they have put higher requirements in orchestration. You can use them in a wide variety of ways. You can do things like programming them with bit streams, which you cannot do with GPUs. So what it requires from OpenStack and Kubernetes is at a higher level. So we'll be looking at that aspect. OK. So if you are an OpenStack user, or an orchestration user, how can you consume any accelerator, whether it's a GPU or an FPGA, a quick assister, whatever? So broadly, we consider different categories of offload first. Let's first look at what you would call offload from your virtualization infrastructure. So let's say you're running KVM on a host, running multiple VMs. The packet flow between the VMs needs to be switched. And it typically has something like an OpenV switch running on the host. And that thing is consuming CPU. The act of switching packets or processing storage for your VMs is consuming CPUs. And from an operator's point of view, that's not really a good thing. Because that's CPU which you could have given to your tenant and charged money for it. So you're actually losing revenue while doing all this overhead. So you want to offload that to save your CPU for what it matters. The other reason or other way you can offload is an application. So from a VM or a container or for that matter, a bare metal process, you may want to offload your computation to a device. And of course, it could be any application whatsoever. It could be machine learning. In a telco context, it could be any NFV workload, like firewalls and load balancers and routers and whatnot. So these are two broad kinds. So in this talk, I'll be focusing mostly on application offload, because that's where we see a lot of the initial interest to begin with. The scope of Cyborg also includes infrastructure offload in the future. But for now, we'll be looking second category. So in terms of that application offload, how can a user consume a device? There are broadly two ways you can do it. One is what you could call as device as a service. From the user's point of view, you're just asking for a device of a particular type, and this wants to use it by himself. So for example, you could say, give me a GPU, which happens to be AMD Radeon 10. And I'll just use it. I'll just program it myself. I'll give me an FPGA, which is an Intel Aria 10. I'll just program my own bit streams. So this is renting out devices. It's using it by herself. But in the context of FPGAs, there's an additional wrinkle. If you let a user program his own bit streams, you don't know what he's going to bring. What's in that bit stream? If you're an operator, they're going to be worried about facts like you could put a malicious bit stream, which may toggle your transistors too fast and probably cause local hot spots and burn your device out, or it could do other bad things potentially. So from the orchestration point of view, we are going to introduce some mitigating factors. There are two models we can do to address a security issue. One is what I call request time programming, which is that when you make a request for a VM with, let's say, 16 vCPUs, 128 GB RAM, whatever, and also, you ask for an FPGA with a particular bit stream ID attached to it. These bit streams could be residing in plans. So you're just using or pulling them over. The infrastructure is doing the programming for you, as opposed to the VM doing it itself. So there is an intermediary, which is your OpenStack infrastructure, which could validate the bit stream in some form. How you do the validation is a very broad subject. I'm not getting into that, but it's at least possible to inject some policies and do some validation. The other broader use case is what we could call runtime programming. A running VM starts asking for bit stream. It says, I want bit stream A now that go and program and start using it. Data says, I want bit stream B now, and so on. Here again, you want the request to be intercepted by the underlying infrastructure so that it can enforce policies and carry out authentication validation. The third possibility is direct programming. We don't even get into that because it freaks operators out. So we'll leave that out for now. The other broad category or other broad way in which a user could consume a device is by its functions. So if you're a user, you really don't care about what model GPU it is and whether it's RER10 with this bit stream and whatnot. You really want to offload GZIP. That's what you care about. Or you want to offload IPsec. So the second model is saying, give me this algorithm or this function. I don't care where it is. So it's a higher level of abstraction. We expect this to appeal to a wider variety of audience compared to the first one. The first one is really for more like power users. It's not totally device-independent because on some level, you've got to say what device drivers you have in your instance image. If you've got a VM with certain GPU drivers, you can only operate GPUs of that particular class, unseen by FPGAs or quick access or whatever. So there is some dependence, but it's at much higher level than the devices service model. But from the operator's perspective, we can meet that use case in two ways. One is what you could call pre-programmed scenario in which all your functions are already programmed, in the case of FPGAs, and just providing them. The user can only see them and kind of consume them. You cannot go and program them. There are a bunch of reasons why you may want to do pre-programming, but that's why. But apart from that, you could also do what we could call orchestration-driven programming in which this is basically the holy grail in the sense that the user, as for a GZIP, orchestration goes about looking for a free GZIP instance somewhere in your cluster. If it doesn't find one, it's going to find a blank FPGA somewhere, reprogram the right bit stream, and then expose it to the user. The user's point of view asks for a function and got it, and all the details are left to orchestration. So we want to include all of these use cases in any orchestration framework, whether it's OpenStack or Kubernetes or whatever. So we'll now look at how we do it in OpenStack. OK. Looking at cyborg architecture now and how we are going to realize these use cases. So cyborg, like I said, is a project for lifecycle management for accelerators in general. It's for various class of devices, like GPUs and FPGAs, et cetera. So we are starting with, of course, the most popular ones, the GPUs and FPGAs, but it could bring other things also in the future. As with any OpenStack project, we aim for this to be vendor-neutral and also hypervisor-neutral. So it's not just for the KVM hosts out there. And there's a link to cyborg here for those who are interested in that. There are more in the reference slides. OK. So why do we need cyborg? So you already have NOVA. You already have GPUs used with NOVA. So why would you not just do more of the same? So here's the reason, right? So with NOVA, use PCI white lists. And as many of you can relate, it's kind of difficult to use. It's not easy to operate. And more importantly, you are not exposing the device properties, per se. They're exposing only PCI-Ds. So for something like GPU, maybe you can get by with that. But for something like an FPGA, the PCI-D really does not tell you what's inside the FPGA. It could be a GZIP. It could be some telco workload like EPC. So just looking at the PCI-D tells you nothing at all. You need something more. You have to go inside the device and find out what function is programmed, or what bitstream is programmed, or what could be programmed into it, et cetera. So this strongly limits what use case can actually realize. And that's what cyborg sets out to solve. So what cyborg does is to have a concept of drivers. I'll get more into that later. But these device drivers, which are provided by the vendors, can go and discover devices and their properties and publish them to NOVA on placement in a standardized way. So a device is discovered in a structured way and represented in a structured way. So it can now kind of have visibility into what's in the device. And that enables all the use case we talked about. OK. So this is a very broad view of what a cyborg deployment looks like. A cyborg could be used standalone, even without NOVA. So here's what the deployment looked like. So the big box is the compute node. You've got a cyborg controller sitting off the one side. And there's an agent running on every compute node. And that agent is augmented by one or more drivers. So you could have one driver for, let's say, AMD GPUs, one driver for Intel FPGAs, one for Xilinx FPGAs, and so on. So these are the vendor-provided drivers. And we also have cyborg drivers to work with them. So the whole apparatus is designed to have all the vendor dependence isolated within the drivers. The cyborg core itself is vendor-neutral. So I'm showing an example of a SRIOV device here. But this is really not a PCI-centric approach, although we'll be starting with PCI devices in the initial implementation. So basically, if you think of a SRIOV device with physical functions and virtual functions, we could take the physical function and use it for management and orchestration. And the virtual functions could be used for a data plane and assigned straight to VMs or containers. You would also need something inside the VM to operate those devices. But it's a higher level of abstraction. So you would need something like a device driver for the appropriate device in there. If it's a GPU, we need an i9-15 driver for Intel, for example. And at the application level, so let's say what you want to do is compression. You're doing GZIP. You could have an application framework like Snappy, which is a general purpose compression framework. And that application framework could then be written on top of your device framework. So it could do the offloads at that level. The higher level application, which uses the framework, could be an agnostic about what's going on underneath. So this provides one level of isolation. But this is really orthogonal to Cyborg. It's only a recommended way of deploying your devices. So in terms of Cyborg structure, it's structured like many other OpenStack projects. There's an API server. It's face the external world. All your requests go through that. There's a conductor also running on the controller. And it talks to a database, which is MySQL, like most of the projects. So this is all fairly straightforward. On the compute node, you've got a daemon called the agent. Somewhat similar to what NOAA compute would do, we would handle devices through Cyborg agent. And the agent has a standardized API to various drivers. So some drivers could be in-tree. Some drivers could be out-of-tree. So you can combine all of them in a given operator's deployment. And each driver will handle a family of devices. So putting it all together in how it all fits in OpenStack. On the controller side, on the left, you would see NOAA glancing neutron. Cyborg gets added to that now. On the compute side, you've got Cyborg agent. And one thing worth noting here is that NOAA talks to Cyborg, but not by directly calling through its rest APIs. It goes through an intermediate library called OSACC. This is based on the same principles as how NOAA talks to Neutron or Cinder today. So NOAA talks to Neutron through OSWIF, it's a library existing today, and to Cinder through OSBrick. And the proposed model here is to use a new library called OSACC to act as an intermediary between NOAA and Cyborg. So Cyborg would return something which is totally neutral and not NOAA specific to be used in standalone use cases. And this OSACC would convert that a NOAA specific form to be consumed with NOAA. So there are various arrows going around here. We'll talk more about that. We'll see how NOAA and Cyborg interact. But before we go into the workflow, we should first look at how do we represent devices in OpenStack? So we already have the notion of placement in NOAA. So with NOAA and placement, we represent compute nodes as resource providers. And the resources they provide are your CPU and memory. And you could represent trades or the properties as compute nodes as trades in placement. So for example, if you've got huge pages or any of your special properties, they become trades. Along the same lines, we want to represent devices as resource providers. And the accelerators within them as resource classes. And this device resource provider will be nested within the compute node resource provider, forming a tree. This notion of nested resource provider has been taking shape in all the past couple of cycles in Queens and Rocky. It's more or less there now. So we intend to leverage that to represent devices. So from a user's point of view, a typical request would just give me four VCPUs and 12 GB of RAM, which look like resources. In additional class, for additional resource classes, like a FPGA accelerator resource class, and the properties would be expressed as trades. So using all your standard NOAA and placement mechanisms and the existing request spec mechanisms, you can phrase your request for accelerators in different ways. So here's a simplified device model. Although it's not that simple when you look at this slide. So basically, the idea is that you've got a device. Within a device, you may have several subcomponents. So in the case of a GPU, it could be just a single component. But an FPGA could actually be divided into multiple regions. You can actually configure each region separately and could contain a separate accelerator. In fact, one region may even contain multiple accelerators. Our definition of an accelerator is a unit of offload, which can be independently assigned to a VM. So a single device for GPU, for example, could contain multiple VGPUs. And each VGPU can be assigned to a different VM. So that would be an example of an accelerator. Similarly, in the FPGA, each region may contain, let's say, a Gzip bitstream, which provides four instances of Gzip. And each of those instances would be a separate accelerator. So we have a generalized model of one device with multiple components, which we call deployables, which correspond to resource providers. And each of them could contain multiple accelerators, which are instances of resource classes. And any property of an individual resource provider, like an FPGA region type or a GPU model type, et cetera, would all be traits in this representation. So to make it more concrete, here's an example of how you could do it with the GPU. So the PCA card containing GPU is what will model as a device in Cyborg. No one doesn't know about it. It's not a placement concept at all. But we have a notion of a device in Cyborg. So you can track things like maybe your BMC or flash capacity in your PCA card or whatever. Anything at the card level will be tracked as a device. The actual GPU itself will be a resource provider. And if you've got VGPUs, there'll be accelerators within that. If you don't have it, there'll be a single accelerator of a GPU class. On your model type, like N-Media, Tesla, AMD, Radeon 10, whatever, that could be a trait. And if you've got a ASIC, like a quick assist, which can implement different functions, then the function also can be expressed as a trait. So we're translating the device components to no-one placement components. And once we do that, now we can start expressing a request, like I said before. GPU as a service would look like this in the left box down below. So in the extra specs or in your flavor, you would ask for a resource class GPU. You're asking one of that. So it's a GPU equals 1. And you're putting a constraint that the resource provider providing that resource class should have a particular trait, which is a particular vendor model combination. You're saying it's a required trait. So this combination means that you're just asking for a device, and you're going to handle it yourself. Whereas if you've got an ASIC, and you want some kind of an accelerated function as a service, you would ask for, let's say, the XYZ ASIC. You want one of that. And you ask for its model to constrain what device drivers you've got in the VM to correspond to that. In addition, you may say that I want this ASIC with a particular function in it. In the case of FPGAs, both types are possible. You could do device as a service as well as function as a service. And it unites both models. So like before, the FPGA PCA could be a device. Regions would be resource providers. And there could be multiple accelerators within them. And every property of FPG of interest would be a trait, et cetera. And the boxes down below would show how you can realize the various use cases using these resources and traits in the context of FPGAs. It's a very similar model. You ask for resources, traits, et cetera. In addition to those, we have a new concept called cyborg properties. So if you look at the bottom right, if you want FPGA as a service, and if you want request time programming, as you talked about earlier, you would express the bitstream ID you want as a cyborg property. It's basically a extra spec with a special type of key value palette, say it's axl, colon, bitstream ID equals whatever. This is not something that's known to know our placement. It just comes with a request, and it's going to be handled by cyborg. But in terms of the user request, they all look united in a common format. So I hope it makes sense so far. If you've got any questions, please do stop me. So like I said, every one of the use cases, the left side shows the accelerator function as a service in a pre-programmed case, an orchestration program case. So we kind of worked out how to realize a workflow for each of these combinations, and how to represent the device as well. OK, here's another new concept. We are familiar with flavors. Flavor is a template for what resources and what kind of a VM you want. But if you put all your device requirements in a flavor, there could be a problem. So you may want various combinations of devices. You may want GPUs of different types. You may want FPVs of different types. For a particular workload, you want eight of a GPU. For another one, we've got only four of those. So there could be all kinds of combinations. So you may wind up having one flavor for every combination of those, and that's a problem. We don't want a flavor explosion. So to solve that problem, we factored out the device requirements into a separate thing called a device profile. It's basically just a flavor for devices at the end of the day. It's going to look very similar to flavors. All it does is kind of split the flavor in two and then let it be composed together at runtime. So a typical example of a device profile looks kind of like this. You can group it together. So each group would ask for a resource with associated traits. The reason for having groups is that you want to have some control over which resource provider satisfies this request. So if you ask for two resource in the same group, they'll come from the same resource provider. If you ask for them in two different groups, they may or may not come from the same provider depending on other considerations. If you know about group policy in the granular resource request syntax, it's the same concept here. So that's a device profile. So how do we put all together? So what's the actual workflow for no one cyber to work together? So at the start of the day, the operator has to know what devices got. So cyber drivers are going to discover the devices and represent them in placement. Once that is done, the operator can create flavors and device profiles. So you could create a device profile for, let's say, GPU small or GPU large, or it could be based on workloads like here's a machine learning workload or here's your compression workload and so on. So you create device profiles for your use case of interest. And then the user asks for a VM with both a flavor and a device profile. So in the initial implementation, the device profile may be folded into the flavor as an additional property in the extra spec. But in subsequent implementations, you may separate them out as two independent concepts. In either case, the user is asking for two different things. It's a flavor and a device profile. And once you do that, as part of the regular NOAA schedule or flow, NOAA is going to first call into cyborg because it does see this device profile name being mentioned somewhere. So it's going to call into cyborg and say, hey, get me what's in the device profile groups. So the request encoder does a device profile's name, but the details of device profile sitting in cyborg's database. So NOAA is going to call into cyborg and get those details. And as part of that, cyborg also creates something called accelerator request objects, or ARQs. These are roughly similar to what you would find in neutron, like a port and a width put together, roughly. Essentially, it tracks the state of a request for an accelerator. So as it goes between NOAA and cyborg in various stages, the state of the ARQ will change. And even the terms we use are broadly similar. We're talking of creating an ARQ and subsequently binding it and attaching it. So let me explain a little bit more. So once you create an ARQ, the NOAA schedule is going to go through the usual process of querying placement and then getting all the allocation candidates, which would now include the devices because they're nested resource providers. So placement is going to return them in the resource provider tree. So now that you've got a list of allocation candidate trees, NOAA schedule is going to pick one. And once that is done, you know which compute node is going to be used. And you also know which device within that specific resource provider within that is going to be used. So at that point, NOAA calls into cyborg and says, buying this ARQ, which means bind it to a particular host or a particular device. So at that point, cyborg is going to go off, talk to the compute node, and maybe program the FPGA and configure the GPU or whatever is necessary. And while it's going on, NOAA goes out in parallel. And it calls it to the compute node, while it's going to work with Neutron and Cinder and so on. At some point, when the instance is about to be launched, the NOAA is going to call into cyborg and make sure all the ARQs are ready. Programming is done, GPU is configured, everything is ready to go. At that point, the NOAA word driver could then attach the accelerators to the VM. So what cyborg returns back to NOAA after binding is essentially what we could call an attach handle, like a PCI-VF, for example, or a mediated device UoID. And that's what gets composed by the word driver into the VM's domain XML. So the basic remedies are create, bind, and attach. And we think these parameters will work for any combination of operations we can think of, at least within this scope, without going to infrastructure offload and such. So for example, if your instance placement fails, you try to spawn the instance and the spawning fails for whatever reason. You want to retry on a different host. Then you would unbind the ARQ on that particular host and re-bind the same ARQs on a different host. Similarly, if you're doing any instance operations like suspend, pause, resume, or any of those combinations, these same parameters should work. And if you've got various failure modes, like something went down somewhere, again, you should be able to use these same parameters to handle all of those cases. No, we would expect them to be the same host. So the device are nested within the compute node. So the resource provider tree, you've got a compute node as a resource provider node. And its child would be a device resource provider. And the placement would run that tree. They'd always go together. OK, so the question is, why does NOVA even need to talk to Cyborg in the first place? Because Cyborg already represented all the devices as providers and resources in placement already. The reason is this. You already represented it. But to actually attach an accelerator, you would need something like a PCI-VF or a media device UID. We also want to program the device in the first place. So for example, as a blank FPGA, we want to apply a gzip bit stream before VM comes up. So you have to let Cyborg have a hook in between. Any other questions? OK. So there are much more detail than what I presented here. There's a whole wiki covering Cyborg. We have many specs currently in progress. There is one overarching spec in the NOVA specs repository, which covers the overall flow. And many of the subsections are sitting as Cyborg specs in the Cyborg repository. We are welcome to look at both. We also have code in progress. So we already have some working code for Cyborg in the Rocky release. We intend to build upon that using this new workflow in the Stein release. And we are always looking for active developers. So if you're interested in this area, please do come and talk to us. Any questions? Yeah. Is there some modifications in the NOVA virtual drivers so that's to facilitate the interface with PCI device drivers? Yes, we would need some changes in the world drivers. The basic change would be that when the Cyborg returns an ARQ with attached handle like a PCI VF, the world driver should be able to process that and use that to attach to the VM. Are there any plans to move, for example, SRIOV handling into Cyborg? Cyborg does handle SRIOV in the initial implementation itself. If we talk of all SRIOV, that's not the plan. For networking devices, it'll still be neutron. With the PCI whitelist handling and so on, for now, at least. What Cyborg is handling is mostly accelerated devices with SRIOV. I have a question. Is it possible to hot-attach the accelerator into a running VM? I'm sorry, can you please repeat that? Just hot-attach. Oh, yes. OK. Hot-attach is possible. It's something I looked at. It may not be implemented right now, but this flow should handle that also. One of the purposes for extracting a device profile out is that in the future, you could say, here's a device profile. That's an existing VM. Go and attach all these accelerators to the running VM. So it's on the roadmap. Yes, it's possible, with the same workflow. OK, thank you. Just a clarification. So let's say that the user wants to perform an accelerated function like the Zzip, as we saw before. Can he or she just get the FPGA device or the GPU device, or he or she has to get the VM plus the device? That's a good question. So what we are looking at right now in this flow is VM plus a device. But for things like infrastructure off-load, where you want to just off-load from, let's say, OVS switch in the host to the FPGA, you may not want a VM at all. You just want to configure the device and assign it to, let's say, something like DPDK in the host or something like that. That is possible in the future, but the similar workflow would not be the same thing. We can still use same parameters, I think, to do infrastructure off-load. Essentially doing a cyborg binding process when you go on program FPGA and such. Instead of saying that here's the handle for a VM, you would install something like a VFIO driver and pass it through to user space where DPDK could use it. Put it back. Yeah, so one more question. What about, let's say, metrics of the images and the drivers for the underneath functions providing the FPGA? Because if we have something like using, actually, the accelerator for the specific device, then we have to provide an image which provides some, let's say, drivers for it. So do we have to provide, as an administrator, do we have to prepare images for each of the accelerators? You would need FPGA bit streams for the accelerators of interest. Sorry, that's not a question, OK? No, no. I mean, the bit stream is one thing, but talking with the bit stream underneath by the system is the other one, right? So you have to have an image which boots on the VM, the operating system, which then talks with this virtual function somehow. So I referring to that. OK, OK, sure. Let's go back to this slide then. So in terms of this slide, yes, you would need something inside the VM which can talk to the accelerators assigned to it. And there could be multiple levels to it. So it could all make it monolithic, but a preferable way to do it is to decompose into different layers. So you could have one layer which is called the VF driver in this model inside the VM. So that just leads to the bare bones of the PCI interaction. You could have a higher level like a device API which leads to the FPGA details or the GPU details and knows about how to interact with that particular thing. And that should expose, ideally, some kind of a standardized API to higher layers for use by, let's say, compression and machine learning and everything. And that would be workload specific. So you would need something workload specific at some point. Yes, you could put the accelerator needs in the VM image. One of the issues there is that you may have multiple groups in your device profile. You could be asking more than one accelerator. And the VM image says x equals z is the property you want. You don't know which accelerator applies to. So there are some things we still need to solve before we get to that. Is there any plan to extend the Ironic? So it reports the capabilities in our FPGA, GPU, and so on as it does introspection? It's a good question, yeah. There is definitely interest in getting accelerators in bar mental cases. We are in a very early phase of interacting with Ironic. We still need to figure out how to do that. So there is no concrete plan yet, but it's definitely area of interest. There are other things I've not gotten into here in terms of how we do bit stream management, what properties you would need in the FPGA bit streams to be used by orchestration, what are those bit stream developers, operators, et cetera. That's one old category. The other category of interest is FPGAs with networking. That's another emerging area. Lot of telco guys are interested in using FPGAs for NFV. So cyber plus neutron is going to be a thing going forward. Somebody asked, cyber plus Ironic is going to be a thing as well. There are people interested in storage off-load and accelerators. So we can expect a lot of development in this area going forward. Any more questions? OK, great. Thank you, everybody. Appreciate it. Thank you.