 Hello, hi everybody Look like it's a full house Let's start this this presentation is about the irony and Actually, it's in the hybrid track basically Hybrid can mean many things but for us at internet hybrid doesn't mean the usual Definition that exists in the open stack world. It means mostly it makes between physical and virtual infrastructure So if you hear me say I breathe it during this presentation I'm talking about physical and virtual and not the private cloud and public cloud although as we'll see with the Physical infrastructure you could deploy a cloud a private cloud and interface our public cloud with it So this talk is Divided in two parts first. We'll have an introduction of ironic kind of basic, but Short and then we'll move on to how ironic is used at internet. So Internet busy is a Public cloud service provider. So we have regions Around the world that already run open stack and I've been for a couple of years Singapore is still not on it's going to be on the beginning of 2016 so these regions are classic open stack deployments and Internet is basically trying to Use open stack not only for public cloud, but for the complete orchestration of the data center So basically internet is going full open stack So the first step is to have open stack regions in all the data centers So there's there's more data centers than what I've shown before that will be the first step So providing like I said a classic public cloud Then we're gonna go with ironic Bermuda everywhere So we'll have we'll be able to provision Bermuda servers in all these facilities and then who knows All we know for now is that we're going full open stack So we're gonna go with probably NFV in the future anything SDC all the good stuff The the goal obviously is to have open stack as a central orchestrator for all these data centers So about the hybrid infrastructure So the hybrid infrastructure as I've said a mix between physical and virtual is something that we do already for a lot of our clients It works well with our current Networking model, which is VLAN and also it's It's well suited for the type of customers we have so big customers big deployments the rather small number of customers and So that allows these these customers like big enterprise customers to Basically extend in the cloud instead of moving right away. For example, if they have like a let's say a Big database back-end maybe Microsoft SQL or whatever they use in the enterprise these days They don't have to move all this to the cloud in one fell swoop. They can just spin up virtual machines For other parts of their infrastructure, let's say web servers or application and keep them at all So they keep these MS SQL cluster for example that they probably have a hard time just keeping up They don't have to move them to the cloud. So That's good So why bare metal why why offer bare metal? Well, we all do provide a high-performance VM. So Some people would say that these these big VMs are enough to to run any kind of infrastructure, but actually I guess there's still good reasons to to go with the With the bare metal for example, like I said the personal beliefs some customers They just want the they think it's not safe or they don't they won't have They don't want to have the 5% performance hit of a virtual machine. So again, we're a service provider So customers is always right, but there's obviously Good reasons to to go bare metal as we've seen in the the talks the keynotes yesterday and today Containers are getting a lot of traction Also, some of these customers that provide the hypervisor infrastructure for example VMware clusters and stuff like that Um Yeah And also for a one of my friend here Sometimes people may want to access for example a GPU or any other hardware devices that are residing in the the bare metal servers, so That makes sense for them Okay, so just so we're all at the the same level regarding irony What does irony do in short it provisions? Bare metal servers, but if we look at these steps as if human was doing them so basically irony will not rack the server for you and connect it but After that it will part of the machine it will install install the operating system Just like a human would with the pixie boot or ISO configured network Configured in the twerk equipment. So that's we're doing that with netman. That's a major Major step in our work towards automating the the data center. So we'll talk more about netman Also, then the the technician would log into the machine configure the networking in the operating system so irony will do that for you and At the end it will give you a root password or SSH key and you can end the IP of the server and you can pay it so Obviously to do all that irony needs other parts of Open stack So this is coming from the The ironic documentation. I just want to say So I've been working with open stack for a while and the documentation is really coming together for example, the ironic documentation is very very impressive, so Could those two ever is working on the documentation? I think it was lacking in the past, but not anymore at least not in the ironing project because it's very nice We can see there that ironing is tightly integrated with the rest of the cloud It talks to Nova Neutron and and Cinder Actually Swift So that so this tight integration is good But there are some some differences mainly the irony KPI is not a user Exposed so a user cannot a user of the cloud cannot tap the irony KPI contrary to the most of the other open stack projects I don't say Yeah, that's it So as I've said irony KPI not exposed to the end user, so that's fundamental Ironic belongs to the service provider in our case. We're we're service providers So all these servers are ours and we simply lend them to customers So so we're the owner of irony this for us and not for a customer Also, I want to point out that Nova provides a complete abstraction of these of Ironics, so basically if we look at the two two lines The two comments there Nova boot. So the only thing that changes is the flavor So but by changing the flavor with the same exact Nova boot command, you'll end up with the virtual machine or bare metal server So that's very powerful. That's a powerful obstruction. I came with a cost But I think it's still a good good decision that was taken So this is a smaller small Shema I made just to help visualize that they the user will interface with open stack at the top layer and The middle layer the service provider layer is actually for For us the service provider. So these two system we have irony and netman will come back later to netman Which is an in-house project at these two they have APIs. They're really similar to the rest of the open stack Ecosystem, but they are reserved to our operation. So internal only and we have the physical layer Which is the actual metal that still exists because sometimes In the open stack summit people are really really high in the cloud But it all comes down to metal at the end electricity and all that stuff So Just want to point out one thing about the irony Python agent. I think the IP is Something that was done by rack space. So I don't know if there's anyone from rack space here Nope, great work for for from these guys. I Think this is what really makes the server part of open stack. Basically, it's it's a RAM disk You boot it and while it's booted. It's not available to the customer It's either for provisioning or decommissioning or doing any kind of stuff you want It's very extensible But while the server is running the this IPA it is part of open stack and you can do with it what you want So you can install the customers image in case of provisioning and you can also do all kind of checks erase all disk after the customer Is not needing the server anymore Upgrade firmware. You can do analysis and stuff like that. So really extensible again a real Real good job from these guys and it's part of the the standard deployment of irony. I believe as of liberty okay, so we Here we have a small warning we're using the cells and It's required and we're currently running kilo. It's really required, but experimental. So yes that it that exists in the open stack world We'll see why it's required It's simply Well, we'll see but the the experimental part is not that bad it breaks a couple of stuff, but It's not that bad. I mean, it's not like it's failing or it's crashing or anything like that It's mostly that you'll lose some usability of For example the flavors you cannot just create flavors in all sales you have to maybe Go in the database for now. I'm sure some of these problems will be fixed, but The Nova team is coming up with sales version two. So Maybe if you're not in a hurry like like we were maybe you could wait for cells V2 So this is a classic sales scheduling I just want to point out. It's a I was jet lagged when I did this. It's very complicated, but it's actually very simple So basically the Nova boot what there's an additional level of scheduling that happens between the cells So like we said previously with the two Nova boot command So it it will see is it the Bermuda or a compute if it's a compute It'll send it to the standard compute cells Which are running the classic filters that we all know RAM filter for example So it will it will look for a node that has enough RAM to fit the virtual machine So that's simple if it's a it's fits a Bermuda server actually the the filters are different So it's looking for exact RAM because you want to have the server that exactly match. What's in the flavor? so Just there we see the difference between the two you cannot have these two filters in the same environment so either you're looking for a No, that has enough space or you're looking for a note that has the exact amount of RAM You're looking for so just because of this little problem. We have to You have to use cells the cells also provide the scalability So that's good thing for service provider, but let's say if I was in a private enterprise trying to have an ironic Cloud I would that would be stuck with cells without needing it so currently I think it's the is the case It may change in the future I'm sure if you're if you have a development team You could maybe mess around with the the scheduling, but then you'll be stuck with this decision later on Actually, if I was trying to deploy ironic in a private environment in an enterprise, I would look at this product Well product is this program from OpenStack. It's called byfrost It enables you to have Ironic deployment that will provision Bermuda servers, but without having to build the whole cloud around it especially Swift for example, that is a requirement for ironic. So So if you want to do a quick deployment of Ironic for a private business, I think that's probably better than the whole thing and As we can see it has a Hardware enrollment to receive CSV file. So we're gonna talk about hardware enrollment in a moment It uses IPA uses config drive. So all the good stuff. It's it's gonna be a real and ironic deployment All right. So a second part Ironic at the internet. So now we're gonna delve a bit more into How we use it What we do with it and the changes we needed to to make so Different sections. There's a inventory management. We're gonna talk about that netman a great product that we came up with and is open source And we'll talk about the limitations that we couldn't Accept at the at the time in turn interface bonding and VLAN trunking to allow customers to have a better experience in terms of reliability with the interface bonding and also in terms of usability with the multiple networks and Section about attached attach which is the unlike feeling All right. So inventory. Yes. So if you have a 30,000 servers, you may need something better than a CSV file I would agree with that statement We're we're lucky at the internet. We have Ubersmith where Smith is an ERP system that already Knows about all the servers we have in all these different facilities It knows about what's in the server knows about what it's connected to so just with these two We're gonna see that this is almost enough to to properly Operate the irony clown it has information about other stuff also as I said, it's a it's an ERP system So it can do in voicing support and stuff like that, but that's our scope So so if we go back to the three layers we had previously we can see that The physical layer any service provider We will have some kind of burning process at the physical layer when you provision the ORAC You will run some kind of software that will do some Diagnostics and stuff like that make sure that everything is connected properly works works well And this will feed Ubersmith with the actual information of what is located where? So that's the first step that that was happening already before we we Wanted to use irony. So what we did was simply build a system that will take this information and push it on the second layer so push it to irony mostly and We'll see what goes So what kind of information the ERP system pushes to the ironic note so basically the ERP system will create an ironic note and Corresponding to the physical server that already exists in the physical world And it will push down information Like like I've said the same information as before what's needed in in irony for to instantiate a flavor So CPU RAM and disk It will also push power information. So the IP address of the power distribution unit And the port to which or ports to which this server is connected So at this point ironic has the information to power on and off the machine And it will also push information regarding how the server is connected on the networking side So management IP for a switch and again the ports So after this enrollment process Ironic is completely independent. So we won't have like a tight coupling to Ubersmith. It's really like we We fill this information manually basically So, I don't know if you can read that But Okay, so this is ironic note show so we can see that we're using the NGAG driver, which is the driver we We developed So the node knows about the PDU and it knows the outlet So with this driver right at this point we can do Ironic node set power state on enough and we will power on enough the server It's far from from being installed in the internet network, but that will be the first step Guarding the network information if we do an ironic port show on the ports That that exists inside this node. We'll see that we have the switch information as well as ports and The way we we do it is not that ironic will interface directly the networking equipment that would be Too too too much for ironic. So we rely on netman. That man is a system that we develop and this this system will Will abstract all the network configuration from ironic. So ironic will simply ask netman for example Please move this server to the provisioning VLAN and that man will do the job And then move it to this or this network and again that man won't take care of all the networking configuration So netman as I've said it was developed in a house. It's open source. So If you guys want to try it if you have needs for it It's written in python. We have a Really good team of developers so and these guys they work with open stack all the time So it's very similar to the rest of open stack. It's not like we're not trying to give you a bunch of pearl scripts here It has a rest API It's it supports a lot of equipment because as a service provider we cannot just We don't have just a couple of switches. We have different switches and routers So that man has to be able to work properly with all of these So it's a good a good piece of software Okay, here's a Graph of what's going on when we boot When we boot Nova or when we try to to get Bermuda server so we have the Nova boot command issued at the top Ironic would ask again like I've said we'll ask netman to move this machine to the provisioning VLAN Netman in turn will turn to the networking gear do its magic and then Ironic will power up The PDU because it knows the information already at this point the server will boot up in the provisioning VLAN It will take it will pixie boot Get the IPA so at this point IP is running on the machine To IPA sends our beat back it will continue doing so until the end so Ironic knows exactly what's going on on the machine all this time It will naturally what we're trying to do is here is boot an image. So it will fetch the image from Swift through glance right in on the on the hard disk and Turn it off and then again Ironic knows that the installation has completed It will simply ask netman to move the server to the tenant networks Network or networks and that man will simply interface the networking gear And at the end we have a final power on where a machine will be booted in the tenant networks properly configured with the config drive and After three or four minutes it will start pinging so so that's it That's it. We we achieve our goal at this point of provisioning a tenant ice or tenant isolated Physical machines to customers Doing this we we got stuck on a couple of limitations that we had to overcome At first Ironic did not have a tenant isolation and we solve this with netman There's a talk at 340 regarding a tenant isolation in Ironic. I want to go there. I want to see how they did it Pretty sure they didn't use netman. So We'll see that It didn't have support for bonding bonding is basically when you have a server that is connected to two different switches to make sure that you know if as a service provider we need to Update the firmware on a switch or something like that mean to make sure that we do not impact the customer So that was a requirement for us and it did not have support for trunking Which also is a requirement for us because we want to be able to give the ability to our customers to have more than one network To this to this server just like they do with virtual machines So Here's the main the main problem we had in Ironic, there's currently a One-on-one relationship between the virtual interest virtual interface and physical interface And that would with the bonding we would end up with one port group And that would mean at some point if we did not modify Ironic That would mean having only one network on the server Which was not at the least at the minimum we needed the two networks and we need more than two actually so we were able to Write this code that these modifications to Ironic to be able to have as much as 10 vif on a port group so a customer can boot with 10 different networks and They will all be trunked to the server. I have a Little drawing here that helps explain so we see the two switches at the top of the rack So this is a rack. I'm sorry for this modeling not very not very pretty, but So a server is connected to the two different switches like I've said So if we need to shut down the switch to do firmware upgrade doesn't impact the customer if this switch or this switch loses It's uplink for some reason to the router Doesn't impact the customer that was very important and then we trunk all the networks that the customer decided in the novel boot or In the interface attach or detach whatever we have to control over this So it will see all these networks connected in the link aggregation So basically standard stuff, but just automated through Ironic one other problem we had is that the the Opening system network configuration So on the left side you have the what it looks like for example for a standard virtual machine So that's not a problem. That's been in the open stack for a really long time But now with the with the bonding we need information about the the VLANs or the provider networks I guess that was not available. So We had to switch to a new format to pass this information to the to the config drive the network info and And it required clouded 2.0 for it to properly configure this the network as To properly reflect what was asked by the customer at boot time so remember We want to make sure that at the end of this instantiation the whole Process is automated at the end the machine has to ping. So this this has to be perfect. So we had to Backport a switch from cloud a patch from cloning it to point. Oh, I think because we're still you and we're still not at 2.0 And then that was about it So this will repeat here for all the networks that the customer asked for and We also had the ability to attach and detach networks from an existing server So just like we would do on the on a virtual machine We can do surface attach pick up a new network and it will I've seen it in horizon. That's an actual screenshot the line will Will just appear so it has a SDM feeling. It's it's pretty good We were I was satisfied with that But we're gonna move. We're gonna do more with that. I think at some point We're gonna try and leverage even more net man. Maybe do And the ability to do create a network for example, so that would be even better So obviously when when you do this net man will do its magic a new line will be added to the trunk a new colorful line, but the problem is We don't have any agent on the operating system at internet So you will have to to modify the network configuration by hand. So basically we'll have to Go back in the in the server and add a new new bonding interface Okay, I want to talk about enrollment a bit. This is something that's I would say I Don't know if it's really important But I wanted to talk about it because I've been working on it So basically there's a difference between there's a major difference between a physical server and any kind of virtual machines is that a Server once it's installed. It's just a server. There's no security group. There's no netting So it doesn't really matter if it was installed Using ironic or maybe it was installed in 95 with the CD-ROM doesn't really matter We can we can import this or enroll this into open stack to accelerate Open stack as the central point of orchestration for the for a customer So let's say customer has a bunch of servers like a hundred servers Install, I don't know how we can still enroll them into ironic And if he does novelist it will see the server and he can spin up new servers And you know, we're moving these guys closer to open stack even without waiting for them to cycle this hardware So how we do this? We use the fake server of ironic So we create the node just like before with the information coming from Ubersmith create the ironic now But using the fake drivers very important because if not when you do an available You're gonna wipe the customer server. That wouldn't be Very good. So we create net neutron parts around it just to again imitate what's what's on the server So all the IP addresses that are there And with the fake driver in place We do a nova boot instance and we pinpoint it to make sure that the the boot The Nova boot will end up right on this particular server And after that we switch back to the real ironic driver So at this point even if the machine was installed in 95 with a CD-ROM if he does novelist The customer will see this machine if he does novel reboot it will reboot the machine Never delete will delete it and even attach detach can can work with that. So the One of the important part is that this was Looks like it's a I don't know It's it's faked or that will cause problem later on but not at all as soon as the customer will do another delete The ironic will take over and the machine will become a real ironic node being deleted by ironic and being put back in the Inventory of ironic and if it's a good hardware and it passes the test, maybe we're gonna Give it to another customer so Yeah, we just want to this is only to point out that there's a lot of Traction in ironic. So a lot of new specs are being approved for the next cycle Some of those are stuff that we We did on our side with because these these specs were not ready or or not ready in time or whatever but The cost of that is that we'll probably have to reintegrate our code using using those to make sure that at some point the code we have is supported by the community and Doesn't just exist in our open stack deployment so again plenty of traction around ironic and Like we've said in we've seen in the keynote yesterday and today the basically the container will probably help irony gain even more traction So after that we're back to the back full circle to our hybrid infrastructure from the start So the only difference it's just like before the only difference is that These customers can now use a central point to spin up new physical server if they need new Hardware new physical hardware or if they need if they need to extend again this this hardware with virtual machines. They can too so That's it. It's a goal here was to to bring these customers into open stack and I think With the virtual and the bare metal side of it. I think we're in good shape to do that That's it Thank you So, I don't know if you have any any questions. There's a mic. I think there's a mic there or if you scream loud enough Yeah, go ahead. Oh Okay, we got it Have you had any issues with Obviously having sort of a an SDN style network on the virtual side and having you know Multiple IP addresses spun up and sort of overwhelming your physical top of rack switches or physical hardware on the bare metal side But that's a good question. But actually there's no SDN even on the virtual side. We were using a ml2 and Vlan so provision provider networks So it's really we're going straight to the metal as quickly as possible because again This is a service provider environment So it's very important for for my team who are supporting the cloud that we put this in the networking team cord as quickly as possible and These guys there They're experts and yeah, so it's it's a giant infrastructure. So it's not it's really not something that we could use open v-switch with or yeah there's already plenty of Hardware there and know how and all that so We picked this networking model for the virtual side maybe two or three years ago and Very satisfied never had any problem again because we go to the metal as quickly as possible and and that helped us Now because yeah, if we went with the full SDN solution, then we would have to to mix the two but the problem with that is the we see the limitation of Vlan the limitation that we all know and Because we have Large customers. It's not so bad But we already know that the now that VXLan is possible with unicast only we're gonna go with the VXLan Probably this is the next major major project for us So we already have a good solution if you go VXLan instead of Vlan you can have the exact same Right thing going on. Yeah Two questions one is do you support see that in eventually the API will be extended to support bonding? By default rather than the way you did it the virtual machine. No the ironic at the other at the middle part. Yes Yes, actually. Yes, I was supposed to put links to the Please have a look at the ironic documentation and you'll see that there's specs and blueprints for exactly that Okay, secondly also when you're bringing up your bare metals Are you doing some health check on the pure bare metal part that it does meet your specs or criteria that you have? Yeah, well Well the the scheduler will take care of that meaning that the flavor that you picked will will be exactly But again, like I said in IPA what's once IPA is running on the server You can do all kind of checks even like overwrite from where you can do it's very extensible So IPA is the key to doing all the dynastics you want question there Yeah, I I have not had any experience with ironic, but my understanding is that when you spin up an ironic instance that It it takes a while because as you referred to you have to wipe the instance So actually sorry that the wiping is happening at the delete stage. So when you delete the the nova instance It will wipe the I make note. So when you boot up, it's already ready to to receive the image So yes, it can take let's say three minutes up to ten minutes depending on the network throughput You have from Swift and you know the necessary writing of the the disk image Okay, so you don't see significant latency like I've seen some blog posts where it says it's like over an hour No, no, this is enough the erasing of the disk can be really long So again, that's happening at the end of the life cycle of the machine But if you know if you have a small number of machine Let's say in the private deployment you have ten servers if you cannot be you know If they're all you can have all ten servers deleting for an hour or something So you have to have enough servers that the rotation doesn't impact delivery Thank you. Okay So you talked about the net man, right? Yeah, I was thinking How is neutron and net man working together? Yeah, well net man and then you try not probably Know link at all actually because net man is is only That man is is setting the configuration on the switch So obviously it needs to know well that man is go is gonna probe neutron at some point Maybe to get the provider ID or the villain ID and then that is passed to Not even sure if that's what's happening. I think it's coming from yeah But it's yeah, but we have developers here maybe but I'm pretty sure there's no There's no link in what you you think meaning Stronglings if there's information going from neutron to ironic it must be Dylan number and that's about it Right, but I think there's a as I've said there's a talk at 340 regarding a tenant isolation So I'd like to know what's going on. I'll be there. Maybe I'll learn something So the net man is talking to the switches. Yes So how is is it compared to the SDN? Yeah, I'm no network expert. So that's why I say SDN like features I don't know if I know that if we had like a real network expert here I will see like a big difference between this and SDN. I have a feeling but For us it basically comes down to this We have these we have this networking model in the data center Vlan and Vlan trunk and switches So we cannot move Simply into a full SDN solution. So I'm still not sure if we if we have the Xlan and we have a Perfect orchestration of this. I'm not even sure it's called SDN I've been wondering all day seriously So I'm sorry. I cannot insert more. Maybe we have a network expert here We can so it will be kind of a duplicate or you know with the SDN you have to adapt to different kinds of switches Specific switches. Yeah, so so basically all all that netman does to these switches is sitting up VLANs subnets and and trunk so so So that's so simple that I believe it's it's not It's not this the end is no split between the control plane data plane or whatever they call it. So Yeah confirmed by the network expert here So, yeah, again because it's only working with Vlan. It's really doing simple stuff. So there's no No, it's the end complication happening. I think So how is hardware failure? Say, you know about fails because of the hardware failure something me in user may not see it because Your schedule will take care of it But how do you keep track of those servers and how often do you see such failures? Yeah, I can see we we see a lot of failures because this this hardware has been Passed through the burning but as it as it's used obviously the servers are going to fail obviously and I Don't see any any difference You have to understand in turn up has been in the process of recycling hardware and installing hardware for customers for four years So it's basically The only difference with this is the automation and the API. So there's no big differences, but No, I think we just have a better control with this because of the IPA where we can at the end of the life cycle Maybe if the disk is failing and the servers the customer says, okay I don't want this server and the old system it could we could miss that for IPA We can have diagnostic and see okay The server would return back to the customer base, but actually there's a failed disc or something and Automating all the checks and just removing it from from the pool. So I think it's an improvement over what was there already Yeah, quick question. Do you know if? Netman will actually be rolled into ironic at some point in the future because that seems like a pre-valuable Yeah, it is to do bear metal switches. It is to us again at 340. I'll be there see Yeah, I would go if that's what they had in mind. I don't think they're gonna talk about that man I just mean do they see this as something that could happen maybe in there and whatever they're developing or Is it something that only us do? Netman again it fits for us because we're using the VLAN model So we'll probably extend the net man to use VXLan and at this point it may be useful for a lot more people But maybe maybe at some point it's gonna be irony can that man VXLan and it's gonna be a reference architecture Who knows good presentation by the way. Thanks So can you expand a little bit into the functions that uber smith does for you? So for example, what I'm actually curious about is if you depend on a particularly well-defined tested validated networking cabling structure Oh, if you do auto discovery of some kind and maybe like net man just pulls those guys to you I'm not sure I can answer that because that would be the burning process. So if we go If we go back So you're talking about when when the burning process is running and the information is putting in is put into a uber smith This is something for another team actually that at internet that could answer that but Like I've said, we're relying on this for us for that has been in place for years So all I know is that at the end. So I start I start here basically at the end All I know is that the info I rely on the information in uber smith and I Don't know if your question is is related to maybe the use of LLDP or something like that to make sure that the port The switch ports are are the right ones something like that Again, that's the most of the position for example like port number five in this switch matches The service of shame you number 35 or yeah, well LDP could really use I don't know if you know this tool And I don't know if it's it's probably leverage into the burning process. I wouldn't know but But I've seen it it is Yeah, that's it and it's another team who takes care of that But I know that there's a blueprint and ironic regarding regarding LDP and I don't remember exactly but again To help if you don't have this process So you can rely on LDP on the server Maybe IPA can run LDP for you It will get information from the switches regarding the switch port of each connections And then you would get the burning process basically and the specs really required to pass from uber smith to Ironic are those standards or something that you modified? No, that's standard what we're looking here. For example, this Imagine a CSV file. Yes, or something no No, no not at all. So basically what when do we the so the second step? Here the step to the enrollment what we do is we tap the ironic API and we push Information into ironic as metadata Nothing special Microphone just beside you So what about if the the clients want to Upgrade memory or there's some half effort So we need to replace the for some of the neck card that Matt address totally change So what about the the burning process? I have to consider the bare metal or a position for the customer So what were you going to do? Well in this case if If the customer just want to upgrade RAM or something like that It's probably something we would not support because if we change the RAM in the server And we return back to the pool then we won't have probably a flavor addressing this this machine So it won't be able to come back. So at some point that would be So on the day you want to the hardware specification has to match to the yeah, exactly Yeah, exactly. So at this point if a customer wants more RAM or something We would probably assist him into booting a new machine with more RAM and okay So what about the replacement of the the hardware like the mad address and like cards change? Yes, yes, the first burning David David Pull the information like Matt address of everything's right. Yeah, they can't fail. I need to be pasted. So So what is happening? It's a good question again. That's more for the operations team. So Okay, but yes for the for the machine to to stay in ironic and at some point We would have to update the ironic node with the new Mac address or something like that. So I think our second question. So Typically in computer, we have an open V switch So when we guess when we know the instant we need to inject the security group, right? Yeah, so this IP table written in the right that the computer, right? So in this I wanted so when we provision the bare metal So where is the security group? There's no security group whatsoever. So how can I control the security for like the pot 22? Well, I either you have a firewall because we offer firewall manage firewall or something like that So have to manage on the level fee rather. Yeah, or are you install IP tables on the machine and do it? So basically again, we are just talking about Exactly like if the customer was asking for server in the old-fashioned way meaning there's no secret So there's no there will never be any anything like security group for bare metal Especially for us at internet. We don't even want to have an agent on the on the server Okay, and there's no since there's no iProvisor underneath. There's nothing much you can do so to address this problem When we when I was talking about moving more and more towards open stack as a central orchestrator at some point We could have This met this firewall this physical firewall is managed also by open stack. So instead of having security groups You could you could say Through the open stack API. I want this IP to be passing through a firewall and I want this this What they call them service chaining Service function chaining SFC Thank you. Awesome. Thank you. Thanks Any more question? Right. Oh Yes Oh, so there's a mic coming in Just one question about this ironic instance and my question come into the snapshot for this ironic instance In case I'm you know planet maintenance activity or something like that. Yeah No, we would offer all the services we offer currently to this the bermadol server So for example, we have managed backups and stuff like that that you could use but At 340, there's another talk in parallel with the one I was talking about It's a mix between Cinder and is it? Yeah, Cinder and yeah, so this is this could be interesting So at some point if you have a sign you could expose it to our ironic and maybe it wouldn't be a Snapshot, but you could you could leverage Cinder to take a copy of some I hope to learn that kind of stuff So I will split in two and go to the boat Thank you