 They will they're gonna do the swap here. So, well, no Hey everyone welcome, I'm Morgan family burger and ptl of Keystone You guys probably saw the keynotes while you're here. I hope You're planning on seeing some other session. You might be in the wrong room. I can't help you there Today, I'm here You know I'm here Jesse long We're gonna do a quick rundown on the technical piece of technical hurdles and everything we ran into for putting together what you saw on stage this morning I'm gonna ramble a little bit here about what what the crazy idea was and how we got here and Then I'm gonna hand it off to Jesse and Guang to talk about that And then we will cover the rough edges and hopefully leave plenty of time for lots and lots of Q&A Because that's probably why a lot of you are here figure out how you can actually make something like this work for yourself So Keystone to Keystone Federation. It's the core of what we have It's where as we described you have a token from your local Keystone you ask for a SAML assertion let's use standards and You then move on you then get your use that SAML assertion to authenticate to the remote service provider And then it works the same as Everywhere else you use OpenStack. Everything looks Pretty much the same So we really had this crazy idea but as you can see You know trying to diagram out how all that stuff works. Well the long the short of it is You end up with a token that's not from the cloud that you started from The crazy idea came to came to mind that we'd love to see a real-world case where Keystone to Keystone Federation solved actual problems that Come that companies have how do I do rapid add of resources? How do I? rapid how do I handle The case where I need a provider in a geographic location that I don't have a data center it's Just it's difficult to it's difficult to do with current with the current open world of state of the art of OpenStack and say Juneau Because you have to find either a provider and then you have to write a whole bunch of glue code to get you into The remote cloud you need to figure out. Oh, I need passwords for each cloud. I need to either use Ansible or Puppet or a whole lot of other things After we sat down and said well, we have this cool thing called that we're calling Keystone Keystone Federation identity Federation We looked around said what industries really do run into this problem television and film clearly have this have these problems live in this world and Clearly it makes a difference to be able to turn on and turn off Extra resources when you're into like pilot season because that's a scary amount of extra work to do I know trying to get meetings with you as a challenge So I really only have one one additional question here And I think that it's a really important one to cover get you know, why are you? Why did you pick open stack versus some of the other to some of the other like options out there? I mean, it's There are a lot of options We wouldn't be here talking to all these wonderful people today if you hadn't picked open stack maybe somebody else, but We've been an Amazon summit Right. Okay. Sure. Sure. Yeah, but what's the why open stack because you did you know use open stack on top of Amazon? Why? Why did I choose open stack? Choosing a cloud solution is sort of like a marriage It's like a relationship and and you try to find a common interest that you both have and For us it was it was being open and it was about about being able to get away from sort of the proprietary infrastructure that Media and entertainment has been stuck in for a long time and and to some degree when we first got involved with open stack Three and a half years ago. It was the promise of federated identity even back them It was the promise of of being able to scale out instantly and and migrate across Providers and not necessarily really locked into one provider or another for for a multitude of reasons So that's why we chose open stack Cool. So I'm gonna now turn us over to Jesse so he can talk about Hosted private cloud and the awesome stuff that blue box did to make this happen Right. So blue box. We are hosted private clouds. Every one of our clients gets their own entire cloud So in this case Yom and digital film tree got their entire cloud to themselves But I think we had maybe the easiest time out of everybody in setting this up because we give them their own cloud We didn't have to implement it on an area that had other people on it All we had to do is stand up the cloud set up the relationship and give them the access and off they went but let's talk a little bit about what was Specifically different in a blue box environment so our clouds One of the options we have with our clouds is hyperconvergence where we're running our Control and our compute in the same place and we're running some of our services a lot of our control services in the same place and We're also terminating SSL with hop proxy and we're doing active active load balancing with hop proxy as well so what that means is our services like Keystone aren't necessarily directly listening for external communication all that's bouncing through hop proxy and That also means that they're running on the same host as some of the other services and what that meant as a challenge Is that the Keystone as a service provider is designed around running behind Apache? And we weren't really running behind Apache for for Keystone at that point. We were running Horizon behind Apache and that was one thing but trying to introduce Keystone behind Apache on the same host that horizon was running behind Apache was somewhat entertaining Particularly because we implement our software installs as virtual environments You we want horizon to exist entirely within its own virtual environment We want Keystone to exist entirely within its own virtual environment Apache WS GI It doesn't really like that idea of running multiple processes in different virtual environments So in this particular case we had to allow Apache to share the Python path of both virtual environments and you were pretty lucky that That the services didn't have too wild of conflicting ideas about what they wanted in their environments And so it worked out pretty well Some of the other challenges were had to do with with HTTPS termination SAML is wants to have The HTTPS come all the way through to Apache and we weren't doing that We were terminating that the SSL at hop proxy before sending it on to Apache So there's a few tweaks we had to make in hop proxy in order to pass all the right things along for SAML to accept what it was getting and a few tweaks on the SAML side as well so that it was Getting it was expecting what it's going to get out of out of hop proxy Yeah, some of those were pretty much just documentation tweaks some of those We were already discussed out in the community. We just had to find them and implement them But that was that was pretty much it. I Think that's Yep Testing Yeah, so We have a little bit of a hard time. So Jesse have an easy time. We have a hard time getting the stuff to work in public cloud Primarily because we are multi-tenant clouds. So that's what means is that we you know share hardwares and you know things like that So we have a lot more requirement to get up and running then just spun off in a separate instance So first of all HP public clouds. So like I said, we have multi-tenant cloud and We have MongoDB back ends which have two regions. So we have East and West and then We have a Mongo web cassette which Running on those two regions and then we have our own mongo drivers. So which means that we are running Keystone From upstream, but we have our own mongo driver back ends and we also run our stuff in Apache So we've been running Apache. So lucky for us unlike Bluebox We've been running Apache since day one when we have Keystone up in public cloud So that's that part is much easier. And then of course it requires a shiplit So that's part of the requirement for Keystone's Keystone Federation. I think that's part of requirement for Keystone's a service provider type of Federation as well And like I said, we are tracking kilo. So what's running in public cloud right now? It's pretty much based on kilo So before we can run this stuff in public cloud So when we deploy a new feature in public cloud, so we have to go through a whole bunch of crap I think one of them is security review, right? So our security You know team got to ask this question. So the base on there, you know threat modeling on this strike modeling like You know things like spoofing tampering and all that stuff, right? So to ask questions like, you know, hey if the identity provider Get compromised, you know What happened, right? So if somebody say intercepts a Semal assertion can the guy replay over and over and get tokens If we don't protect the communication link, you know, what happened and how do we set up trust? You know all these kind of stuff So they asked us a bunch of questions as turns out there was a bug that was pretty critical that we fixed Which is 104 2916 Be like Adam, you know That bug is that we basically had to you know without that fix we would have to turn off signature validation which Which is that good So we basically had insecure SAML assertions like a non validated SAML assertions. Yes So yeah, so we had to fix this bug before we can make it into production That bugs had to do with you know, the way we treat namespace in in XML So if you know the way we do, you know, we sign the the samples that we use this to call XML SCC one and We sign the stuff and then we basically bundle that up and then when we on the other side when we validate that Turns out, you know with XML library if you don't explicitly namespace this stuff We just assign you an arbitrary namespace and that basically invalidated the signature so we have to fix that bug and Also, unlike private cloud with with the with the public cloud we you know identity data is Tied to accounting. All right, so when this keystone keystone Federation feature first Introduced it was based on this Greek identities. So which means is that you can map an identity to a group, right? So that identity doesn't necessarily need to exist in the in the service provider But with public cloud that's that's different because we need To be able to map an identity to an account so we can meter on that identity and be able to build the guy right on the usage You know things like that. So luckily, you know We got some help from from American I CERN basically You know introduce this map to a direct identity feature Which we can take advantage of Yeah, yeah, absolutely So, you know like Jesse touch on earlier So we have to make network changes like we know we have a load balancer. So we have to make it So that the session sticks, right? So we have session affinity and then we also have to make sure we preserve the request URL turns out Shiplet has they validate this binding. What basically means is at the time when you create the signature You have to know what the request URL is that basically bind to an endpoint So if you route through a bunch of proxies if you you know, if you don't preserve that endpoint Shiplet is gonna give you an invalid signature. So we have to make some, you know network changes to make sure we preserve the request URL and Identity management for public cloud is a little bit Involved because it involves right now when you create a new identity is a both configuration and administration changes So the lines basically blur between configuration administration and administration to me is just API, right? Configuration it requires a bunch of stuff like, you know, you know, you write chef recipes and public whatever So for us to deploy this feature we have to you know, write bunch of chef recipes Create the data bags to push the the the meta XML file that contains the The public key for for to validate the signatures and Also administration for us, you know Right now if you want to create a new identity provider create a new map Those have to be done with someone with L3 support privilege and above and those API has to be Audited right so it means nobody can just go in and mess with the map. What do you mean by L3 support? So what is L3? L3 is basically guys like us The guys that know deep into the code and all that good stuff Yeah Yeah, so, you know when you yeah, so when you have problem with HP public cloud you pick up the phone, you know someone You know probably L1 L2 guy would pick up the phone and they'll try to troubleshoot it And they can't do it. They call up the L3 guys and you know, so we go and then and then Look at a log and all that but even looking at the log itself or because we have to deal with compliance We can't just log into the box and we have to request permissions and all keystrokes are logged, you know, that's all that good stuff So all the changes are fully audited so you can't just change the map because you know One of the dangerous thing in in Federation keystone keys keystone federations If you don't have the map set up correctly, you could you know accidentally map to a different account, you know Which you can get some freebies from those accounts So we have to make sure those are correct Yeah, also for public cloud whenever we deploy a new feature in the public cloud We have to make sure you know everything still works correctly, you know make sure our performance is good and basically Simpli is a different process that we introduce. We have to make sure we monitoring that process You know make sure if we you know doesn't chew up a whole lot of resource in a note We have to make sure our regression test is still look good and you know rate limiting Documentations and support runbook, you know the whole bunch of stuff that we have to do before we can introduce a new feature in the into a public cloud so runbook is something like you know if shiplet, you know You know decided to a tandem, you know, what what what can I support guys do right? So we go reboot it or so, you know, how do we safely reboot that box things like that? So we have to document those So that's pretty much What we did so You know, there's a lot of there are a lot of rough edges on identity Federation Let we'll be up front and say that The really cool thing is that they're just they're just rough and easy polished. We have a lot of wish list items We would really like to fix before You know we tell the Pearson who's just getting into open stack and just barely scratching the surface That it's really easy to do this. Well, it wasn't really easy to do this It's quite doable and none of the rough edges prevent this from being a reality and that's the really cool part But you know, let's be fair We're not really chasing down a rabbit hole here It's the Cheshire cat that's gonna come after us But here are all the things that no one really would tell you about unless you happen to have done a project like this And I'm gonna let Jesse and Gwang go over the variety of wish list items and rough edges So doing our senior VP keynote so people say, you know, he wants to be able to run a open stack without a PhD degree So this is PhD education right So so some of these basically That we can do to make it better things like I said, you know earlier we have this You know the fine line between configuration and administration, right? So right now to be able to deploy Keystone keys to Keystone Federation is both so Ideally we want this to be a purely administrative function Which means that whenever we need to create an IDP we should be do that all through API's create IDP Create the protocol create a mapping all down to API's and any things would just work, right? So this is a similar would similar to how you know, where we moving toward in the per Domain specific driver configurations. So we need to store these IDP data into into sequel backend Yeah from our point of view on this one it meant that All of our configuration is driven by Ansible playbooks And we have published Ansible roles that are all open source But in order to implement this type of feature we had to start leaking not leaking, but we had to start faking customer data into the public Playbooks that would get filled in later by site specific configuration But we had to couch it in a way that you could have one Provider you could have ten providers We have to be able to loop over them and have a bunch of stub things in there that if you actually ran it without Providing a real provider you could get some things on your file system that you might not want and your keystone may not work So we'd really really like to see that Site specific that customer specific type of stuff stay out of the system configuration instead be data that goes into the database Once it's all set up, right? Yeah, and then death the other thing for us Which you know is like whatever we make configuration changes our security team team get freaked out Right so which means that you know say hey, what what do you change and so we have to explain to them You know what what configuration changes are and you know, how does that impact security and So so that's you know, we have to go through all internal reviews and internal audits and so that's that's not fun Okay, so second, you know because of this Shiplet You know This this binding validation in in shiplet what we found out is you know shiplet I'm not sure you guys familiar with that. So it has this XML policy that you can set up on, you know, how to validate the You know the IDPs and how to validate the signatures and all that and one of them is validate the binding Which is you know according to that documentation It can't be done through policies But what we found out is it's not the case because the policy validation happened Prior to the binding validation, which means is that it's hard-coded So so we would we don't have a whole lot of choice, but to you know to It's just like endpoint binding right let's let's say you sign a Samo, let's say I will sign something for you Exactly for the particular endpoint and that's that endpoint is part of signature So doing that validation if your endpoint does not match to request URL you got a signature in about in ballot So if if we were to do a cloud with multiple regions that some of our customers may want that means that we have to create Relationships between each region and the service or the identity provider rather than having one to a complete vendor That we could assert multiples in Right So yeah, so so we are looking at some other alternatives I mean the second point is basically you know when it like I think I touch on that part earlier Is that when you put a new service in there, you know all kinds of stuff have to happen We need to monitor it, you know We want to make sure you know our performance is still good and then we make sure it's footprint And you know it's it's still within you know our tolerance level and make sure it's you know Available on the platforms that that we support right and so that's you know Introduce a new service on the service looks pretty simple, but you know for us it's quite a bit of work From our side I mean from the public cloud side the the customers of public cloud aren't necessarily seeing what the infrastructure to run the public cloud costs So if they needed more to run Shibboleth on their keystones They might be able to launch more keystones or make bigger keystones on our customers They get the entire stack and their capacity of their stack as it is a little bit of it's eaten up by the control services We run so the more control services be run the less Capacity the customer has for the stack so that's a value proposition where if we introduce this service They now get less cloud for their money, and that's a big concern for us Okay, so yeah, so this is mostly my complaint is that IDP should be owned by domain right because Yeah, so because you know for us self-service is very important So because when you sign up with a public cloud it's people of cloud we create your domain You are the master of your own domain You can you know you have all in this your own administrative domain By that you should be able to say hey, you know my users You know for cloud bursting myself things say hey my use my users manage in my own private cloud I should be able to you know hook up that through You know through through through the Federation So right now you can't do it right now you if you sign up in HP You want your own IDP you would have pretty much have to call call up our support and we'll file take it and we'll basically have to You know Okay, that's that's the second point right here mapping right now. We only validate the syntax We don't validate the content which means is that you know I? Can I can set a map that map to somebody else's domain right so there's no way no easy way for me to validate that today was The question was is the endpoint validation a reason that you have a hard time federating to somebody you to into a provider that You know arbitrary, but the user are really what's yours? Yeah tequistus No that that the endpoint validation piece. That's just a it's a configuration It's a piece of configuration comes along with how you run shibboleth and how it validates It's just the nature of a SAML assertion. It has to be targeted at the endpoint What's the problem here is that we don't we're right now? We don't have a way of we validate that syntax of the map the mapping is the bundle of attributes in sample in the SAML Assertion to something useful to Keystone We validate that the syntax of that mapping. It's a declarative It's a declarative of like a DSL type thing that we wrote specifically for this this set of the set of tools, but It doesn't validate what the result would be we don't have Given this assertion. What am I going to get the other end at the moment? You you have to have a lot of eyes looking at and try things Yeah, that's it could map to something completely wrong But for the most but but we can guarantee the syntax is right, which is again. It's a rough edge It doesn't mean that you're never that you can't use it It's we should fix we should make this better Yeah The the safe error is that you map to identities that don't exist and the bad error is the map to identities That do exist that aren't yours and now you're able to you're now able able to either Create instances under their account and charge them or worse read their instances and get access to their content Which is never a good thing Yeah, so so this one, you know, you know just put it here But I mean to be fair right right now if you know this You know pre-existing relationship between IDP and SP's through a you know metafile But if an account has been suspended at the IDP, how does the SP know to you know to invalidate that token, right? Yeah There's no revocation, yeah Yeah, so you wanted to have a real-time revocation you would have to build some workflow on top of that So, you know just like the way we have the same problem with LDAP today So if somebody Some user status change in LDAP, you know keys don't know about it Yeah, so The last thing is that key, you know identity Federation is good, right? I mean we solve a lot of a lot of issues with that But the next step what's the next step, right? I mean how do we how we do this complete solution of cloud Federation? I mean, how do we have a global catalog? How do we discover resource and how do we make these workflow process more smoother? I think that's that's something that it's That we'll need to take a take a look at yeah for the demo. I mean, it's it's really we On the blue box side because everybody's a full admin on the cloud All we really had to do is give them map DFT into an admin user and they could create the flavors as they want they can create the networks So they want to create the images that they want that's much harder to do in a public cloud environment But what what we really want is for Digital film tree to be able to do that within their domain so that all they have to do is a stab all the providers have to do is Establish that relationship and then step away and they the the identity provider the source of that information can push All that they want for all those configurable tweakable things push that from their central place out to the endpoints And then everything looks the same to them. It's a it's a happy accident. They're not happy acts but it's a result of some process that it works well In the in the demo, but in reality it would be a much better story to push Yeah So that's so a couple of things from our end the tooling for the service provider It's it's pretty rough right now There there are a few things where you have to copy and paste in Python code and poke at the keystone v3 API directly in order to get some of the mapping in the configuration done Which is difficult to to administrate difficult to automate It's it's not something we want to carry forward into a production environment Also, it involves XML written to a file system and nobody likes that And then we're we would like to see better visibility into the federated usage in our in in the public cloud's case They're mapped to an actual user that exists and that user is getting tracked for all the usage in the blue box side There is no user. There's just a Project and there's just roles or groups within that project But as they were using the cloud we couldn't really see that they were using the cloud and that from from a Showback or from a tracking that are they are they capable of using it? You know, that's not visible to us and we'd really like to see that and before we take it into production as well Yeah, I think I think the IBM folks have a nice demo on horizon side and how to use it You know keystone keystone Federation earlier, so I think that those patches may land pretty soon awesome so With with federated identity we we are a we are able to build federations and we've been doing that for Probably something like 10 years now And and the typical use case is you give access to a resource That to someone that doesn't own the resource We have several federations we can do the same with open ID connect so my question is can can you talk to the relative merits of Doing keystone to keystone federation as opposed to doing it on top of open stack and consider the different open stack Deployments as as separate Relying parties that you can give other people access to Let me think about that for a second I'll throw something out here, so The reason why you might want to do keystone to keystone is particular is specific to Bursting from one cloud to another so keystone has the flexibility that you can back end it with your own Internal identity provider you can back in with LDAP. That's that's your controls. You have everything like that so you've already got this this Your local keystone your local cloud backing into your identity pool What you don't want to do is try and create a connection between Somebody else's cloud somebody else's administrating into your identity source That's not a relationship that you want what you want to be able to do is take your keystone that you're configuring let it Talk to another keystone the other side doesn't have any access to your stuff So it's keystone talking to keystone. It's a it's a communication protocol that they can understand that and as clouds move you you have the API version compatibility going on The HP cloud was it was kilo we were Juno Digital film tree what was it was kilo and it didn't matter because the the protocols were set up in a way that they were able to Talk to each other correctly, so it's it's really about a matter of compatibility And there's a second use case that's very related what happens is you have a You have digital film tree and they have a variety of people who are consuming a service and they each of those each of those Consumers have their own identity provider. Well, do you do you really want to in this case have to have you know 15 identity for 15 people 15 different organizations set up the mapping that the the trust relationship between now HP And then you change and you add blue box now. We have to set up 15 more times. It's really about Making it a little bit a little bit easier a little bit better experience or On the flip side of it making it a little bit more opaque when you don't want that to be visible Yeah, I would say yeah just to add to that. Yeah, I mean we were we're a Service that has clients that is then a client of other services. So so yeah the the How transparent you want that to be or how opaque you want that to be is is key to how we do business So yeah, I would support that absolutely Okay, I'm I have a Few questions and comments. I will first start with the comment you said that you were thinking of different way to integrate with SAML 2 and Maybe you search for a simple SAML PHP. I don't know if you know this product, but it's developed by the R&E community Okay, so that wasn't Okay, so my question It's worth bringing up. Thank you. It's good to know it's good to know it And we will keep a look on any of those options because maybe something will be better mod-off melon is one of the ones Yeah, lots of options Okay, the the question I had was with Shiblett For the moment, I'm not specialist in terms of Federation, but my colleague is and and he Basically said that those there will the Shiblett version 2 if I recall correctly will be end of support And it's been extended a few days ago or something like that to 12 additional months But what what is the plan in the future? So you saw us mention us. I wasn't even standing up there. I'm Adam Young work for Red Hat and I'm Keystone core When we were evaluating the different ways of implementing Shiblett There were a couple things that stuck out and the biggest one for my opinion was the fact that I had to do something Different on the client side than on the server side and there's a library out there called lasso That we could use on both sides and mod melon is in a patchy module that does that now mod melon also Does not require spooling out to another service to do the Shiblett stuff So I really like the idea of mod melon Last so and then that being the the groundwork of our strategy there So I think it's going to be something along those lines having said that and on the on that note A lot of what we're tied to in Shiblett is tied to the patchy modules that are available in distributions once the distribution moves to the New stuff that the later versions that are that aren't near end of life We will obviously be moving with them and making sure we support a lot of it was This was developed on trustee trustee has very specific Acquirements and same thing you know and and support of sentos and rel I mean, there's just specific versions that are out there and of a distribution for lock We're locked to really what the distributions have in this case because it's outside of open stack It is a service that is available via the distributions long term Of course we will move So a quick question and maybe you get maybe this is a bigger question if I have If I have multi-region and so let's say I have three Separate open-stack installs. Have you guys ever considered like the ability to confederate within an identity? So if I want to establish a trust relationship between say partner a my organization I've got three different installs I confederate have one relationship initially and those other installs are are basically already associated with One of the keystone boxes as opposed to if I have you know eight different sites And then I'm setting up point-to-point or separate federations with each of those organizations Yeah, I mean you should be able to do that with multiple regions because I mean it depends on how you set up the Providers you can have a chain of providers Configure in in shipments. Remember what you said about a mapping to an existing identity once you have that mapping setup once You can map multiple federations to that same identity come in by different maps, right to the same identity and do a single identity Okay, you can do it. You don't have to do it So a question with with with regards to how keystone and Amazon kind of could confederate I mean have you guys considered? exposing identity from keystone to Amazon and I don't know if it even makes sense But but what about vice versa, right? I mean could you take I am identities from Amazon and make that available on an open stack Right, we have considered making keystone a full-fledged IDP Let's we have we chose to avoid doing so or both simplicity sake and that we weren't sure that was the right choice There are many many extra things to come along with making keystone a full-fledged IDP It theoretically you could probably do it today. It would be a little stubby and it has some limitations It's not off the table. We're definitely open to considering these types of things Please come to maybe one of the one of the work sessions and talk with the keystone development team and explain You know and help us understanding the your specific use case in that for that and we'll consider it. I mean with the right amount with the right type of demand the Clear benefits will work that and in reverse You know as you said with the I am It same kind of thing. It's help us understand the use cases help us understand What the clear benefits are and we're not going to say no to that should be a different just another federation thing I think it's one that we should deliberately deliberately look at but I don't know what I am exposes as far as Federation protocols if it did sammel then we could potentially make it work right now We'd have to see what what the modules are if it's not sammel and how what the path would be to do that But mostly it's a lot of it like oh, I use using oidc or sammel really at that point It's a configuration and using the right module in Apache to give the attributes out Okay, it's a more Technical question about how to get things you said that you mapped an IDP to a single account for the HP public cloud And I would like to understand. How did you plan to do their role-based access control? are all our members of an IDP Only members for the for a single account or do you have different roles like for example an administrator and sub roles or something like that? Yes, did you plan? Yeah, we have those in tree clouds. So when you sign up with a tree cloud We created a domain for you So you basically have your own administrative domain and then you can create domain specific roles So so you know you can Assign roles to to the to the users in your own domain But that's different from from federation because once once Federation is about how do you you know establish a trust and once you just establish a trust You basically fall into the service providers administrative domain So you can set up the map says hey, I want to map it to the user and then the user can you know What about access that user have is totally controlled by the service provider, right? well, yeah The service per the identity provider will push down to you what the user is and then on the service provider side That's where the mapping comes in of these particular users map into this role or these You can define groups or you can define Rejects is for how you want to map the user identity that's coming in into what rights they have on the cloud itself And that's a service provider level configuration The service provider is authoritative for author for the authorization decisions that's that that comes from service provider No, but that's Can we have time for one more question Steve the floor recognizes the gentleman from IBM You're really gonna let Steve ask a question This is totally your go ahead I Just wanted to know a little bit more about How HP public cloud is deploying keystone under Apache and with the modules With that like modshib and stuff like that, right? So we basically Deploy Yeah, with with Apache with the shiplit and with my shiplit What do you mean? Oh, we have our own ship recipes that we basically we basically created the data bags for the The meta XML file for example, and we have recipes for the for the configuration file changes Yeah, I hope so Well, thank you very much. I think we're at the end here appreciate you coming and There is I'm we got one more one quick thing There's a white paper and I'm gonna hand this off so that you can get word the address Hi, there's a white paper with them It's pretty high level of about the background of the technologies used in the camera to couch demo It's on open stack org slash enterprise under workload portability. Thank you