 Okay, we get started great so welcome everyone to this session. This is a panel discussion on software to find storage in OpenStack So if you're here to listen to that you're in the right place We're we have a fairly distinguished panel with a lot of years of Expertise not only in storage, but in OpenStack So I'm gonna have some questions that we're gonna kind of go through as a panel But we also have mics set up so that near the end I will have time for you to come up and walk up to the mic and ask the panel some questions so Good place for us to start. I think it's absolutely do some introductions. So my name is Ken Hoi I am I used to be at rack space on the OpenStack team there Currently I am at EMC as a business development manager for their OpenStack Cloud Solutions business group so and I will be modeling today's panel So when we go ahead and have the panels listen to themselves. So I'm a Albaran I'm I was previously at Red Hat managing the cloud storage engineering group and today. I'm cloud CTO in Huawei I'm Mike Perez. I'm the project technical lead for sender for the key to release and I work at Deterra My name is Sage Weil I was previously at ink tank co-founder working on Ceph and most now I'm at Red Hat Hi, my name is Shimada here. I'm with EMC in the office of CTO working as a cloud architect And I'm John Griffith. I work at solid fire. I Worked on sender for the past few years. I used to be the PTO and it's now turn over to Mike Thanks, gentlemen. So just to kick things off I think You know in our in previous from previous summits and also in our prep for this panel one of things I think we can agree on is there is no one single definition that everyone agrees on About what software defined storage is let alone what it is in OpenStack So maybe a quick way to kick off the panel discussion is to have you each of the panels talk about how would you quickly? How would you define software defined storage and how would you define it in the context of a cloud platform like OpenStack? Yeah, go down the line Well in my view there are several characteristics that SDS needs to meet One of them of course is separation of control and data plane and a lot of the contention is about what that means So control plane has three elements one is the configuration The other one is the element layout and the third one is the storage services and The question is where do you draw the line to you? Do you separate all of these three or just the configuration the second? Characteristic is programmable APIs. So you want to automate and drive storage policies from the API and The third one which is probably also contentious the ability to run on commodity servers Thank you So for myself Very much just thinking the software defined storage is leveraging virtualization to create an abstract between the data and hardware so that we can have virtualized representations of That storage hardware so that I can have a pooling of dissimilar storage systems and Into a cohesive centrally managed Representation of the available resources regardless of the underlying hardware and as well as the System platforms themselves or their physical location I think that the problem with soft the term software to find storage is it doesn't define what storage means Aside from the fact that it's not hardware. So everybody at least agrees on that I think you can some people define it to be the Software that makes a storage endpoint appear that you can then interact with So you can sort of plug in existing boxes Other people define it to be just actual guts of the storage system that are handling the replication of data across nodes and making it reliable and redundant and so forth It's probably not the actual bits on a disc because that's that's not software I think that the key thing though is in any sort of discussion about software to find storage It probably makes sense to sort of leave the term behind and try to talk about what it is You're actually talking about whether it's a unified control playing Which is you know useful or it's the software that's actually Managing the storage of your data the actual guts of the system Which is also pretty valuable and then asking the question about what that's actually providing you You know what what freedoms is it giving you in terms of freedom from hardware vendors or software vendors or software freedoms in general? Yeah, I mean I definitely agree that the term is kind of vague So we need to kind of go beyond the term and into the actual what it means and to the points Everyone else raises states basically data and control plane so from a control plane perspective It's about Abstraction and being able to provide some level of interface regardless of what the actual on the element management is for the underlying Array that you're trying to control in a sense So that's the control side and then on the data side You've got the aspect of you know pulling hardware whether it's commodity whether it's not but basically being able to have some sort of data Playing initiative to effectively create a virtual array or as you called it last time I think software based storage on top of Hardware of any type effectively Yeah, so this is kind of interesting. I've been very vocal about my opinions on this over the past couple of years The thing that's cool though is we all kind of seem to be converging a little bit so I've I've felt that it was At its core separation of control path and data path right and that was pretty much the end of my definition It was pretty broad I think we're kind of getting closer to being on the same page. So that's kind of cool So let's talk about what is you know, so you had now several different Definitions of what software to find storage is what do you think it's the value that it brings? Why is it important to think about that particularly again in the open stack context? Anyone so So from my perspective, you know, I look at it in a number of ways One of the biggest values and the most important criteria is probably automation in the API And flexibility and then we start getting into the other things people talked about virtualization of the resources and stuff like that Those are the key components and the reason I think their key components is that's the only way you can effectively make things like Open stack, you know cloud computing new generation data center type stuff That's the only way you can make it work. The whole point is we want to get away from you know The the intense proprietary management and things like that that you have to do with these products and so on and so forth In some ways you could view Cinder is doing that job for you At the same time though Cinder does it much much better if you have a product on the back end that actually implements those characteristics Right, so that's kind of in my opinion. That's kind of in a nutshell Why I think it's important Yeah, definitely and it basically allows a level It provides a means to provide an API that you don't have to know the underlying storage plumbing or be generally You know an expert in the storage aspect to be able to reap the benefits of storage within within the cloud itself So the API provides value in that aspect of consumption, but then on the data plane side of definition as well There's value to be had of you know Converged scaling between compute and storage resources as well for the cloud. So that that's a benefit on the data side. I think I Think for me the definition really comes down to freedom So if you sort of start at the top of the stack and you're talking just about control planes having that sort of unified integration into whatever Software orchestration stack you have means you can plug in any vendor solution So you have the freedom to choose between you know existing proprietary vendors or open source vendors or whatever you want And as you move down the stack you you get other sorts of freedom So for the data plane for the actual guts of the storage system, then you can have Independence from your hardware vendors so you can buy your hardware from any white box Whatever vendor that you want and if it's open source software Then you also have the freedom where you actually don't even have to swap out the storage system itself You can just choose which company you want to support that software Which I think is sort of the ultimate or perhaps support it yourself, which I think is the ultimate You know Realization I guess of that that software freedom So I think in any sort of conversation about this you for me, it's always a question of What is what is the solution and is it actually giving me more or less freedom in that context both from from the vendors and From the hardware perspective and from the ability to swap out different software Reiterating those points, but also the point that I made earlier in the definition of having central place for of a bunch of dissimilar storage systems and Having that idea of being able to in a sense Be able to have one pool storage of just dissimilar storage systems But so that's from the operators perspective on the end user. I mean Really, I think the end user doesn't necessarily care what the back end really looks like they just want resources readily available So that they could get back to what they were doing So just add one more aspect and the reason you need programmable APIs is because you need to deploy a lot more Storage and you need to scale up and scale out sorry a lot more than before and Part of that the economics are such that classic storage arrays Just cost too much. So that's part of the commodity commodity Detailization of storage took me a while And that's why you'd like to run on x86 and also simplify your storage. So Part of it is that you don't want too many types of storage in the back end You actually want for simplicity sake to choose one of your choice. So that goes to freedom and scale out with it Supposedly infinitely So based on some of the things you've said it Someone could argue that you should all you're doing is defining what sender is So I guess the question is is sender in fact all there is to say about software to find storage Or is there elements in that are missing within sender? That either you can put into sender or that a vendor could add and Implement plug into sender to to initiate implement other aspects of SDS It parts on anyone so Sender does abstract the the different vendor APIs and Provides a single API that the entire industry can use and program against what it does not provide is the layout and Storage services themselves. So just using sender does not give you that scalability does not give you those capabilities You still need something on the back end that can provide that Whether or not sender should go there. That's a different question not sure Anyone else thoughts So I think that to a certain degree. Yes, sender actually is providing Part of what we called SDS and to the other question. Yes, there's a lot more that we can do and should do The second part is is whether that should be provided from a plug-in My answer to that is no because then that kind of defeats the purpose But what what needs to happen is those devices that you plug in is back ends They have to have the virtualization and automation capabilities so that you can actually expose that through sender So one of the things that holds sender back right now is the fact that we do have legacy storage systems in there They don't know how to do things like virtualization and good automation and stuff like that So we can't get everything up to the same speed that we would like it to be And that's kind of a problem and it's something that we're trying to address and I think we're gonna fix And we've definitely got plans in kilo to make that better So when you say it trust what does that mean? That's I mean you talk. Yeah, it sounds almost like you're saying Well those legacy storage can't do SDS and sender So therefore we're gonna bypass them and either create it ourselves within sender or use some other technology We're gonna fake it So what we're gonna do is we're gonna help some of those guys that are further down and back in the old legacy world and stuff like that I can't do certain things We're gonna help them fake it inside of the sender code and make sender a place where we can actually Try and pretend that they actually work the way that all play together Yeah, wait, what's some of the folks on the panel? Is that a good idea? Is that a is that a bad idea? What are your thoughts on that I think it seems like it's generally a good idea I mean cinder is definitely solving that control frame problem right where you have a single interface that lets you orchestrate these other systems I think that it seems like it's a it's a it's a sliding scale though like you can you either pick the most like Generic common denominator of all the systems and you get something. That's only modestly useful But every vendor can plug into and call themselves and software to find which is which is Where we are today Or you can see you have at some point you're gonna have to pick pick a stopping point like you you you're gonna have some apis and only certain systems implement and others won't and so it's There's no I think there's no silver bullet, but I think I would assume that everybody sort of agrees that cinder is mainly Concerned with that that control plan right. It's not actually a storage. It's not you're not using cinder software to define the actual Right, and I don't think there's much interest in so yeah And to that point there is a reference implementation that uses LVM That is the probably lowest common denominator, right? I would love to see that improve and get better Or you know or whether it's replaced with something else is the reference. That's fine But yeah, I think to your point, but then if you want to fake it as you said right you have to provide those capabilities so it's either You do provide a default implementation and use either use an architecture like open GL where You have a soft cinder software implementation and offload to storage where possible Because as under the assumption that your storage if it provides these capabilities will do them better or you do a Completely new implementation But you can't fake it otherwise, right? It's so so we do things today where we fake it Right, we we can fake things inside of cinder to make devices that don't actually support things actually work We can pretend that a device has the notion of a pool, right? So then that way devices that do use pools. Sorry for them They can actually still work and function and get value extracted and everything else. So so that's one example The other thing to keep in mind is is there's a difference between Faking it in a driver and faking it in the top layer inside of cinder and the main point I was trying to get at was What's needed inside of cinder more than anything else is Smarter scheduling capabilities smarter, you know more hand-on You know fine-grain control. That's what's needed The problem and the reason why that's hard in an open-stack context Especially is because there are so many back-end products out there that don't have those capabilities They don't expose that sort of functionality to you. So which is where a default implementation would come in handy, right? How do you pick the right one? You don't it's it's just an implementation It's not the best if you want the best you probably buy the best of breed on your underlying storage You don't expect To get you know to have your cake and eat it too Sure to an extent. Yeah, I'm not sure I follow completely well if you do provide a full implementation Right, but you only use it when the underlying storage does not comply with the needs sure right then You've solved the 80 20 and the rest that want something very highly, you know With high performance, etc. They will buy the hardware that provides those capabilities Yeah, and maybe this is where it also comes into play of you know what data services you expose and what the use cases that you're Supporting because again, that's that's another spectrum that everything varies on as well along with the performance and virtualization aspects So we've got a factor that into and then maybe that's where the center minimum API comes into effect as well So most of you know, I think they are they have been Co-submitted to buy vendors various vendors not just one to essentially Have that functionality you've talked about that you want to put in the center as part of a vendors plug-in So is the message that you're advocating that those vendors shouldn't do that anymore They should just submit all the code back in descender or what is the message for those vendors? And how they want it, you know, because they know the answer to that Why would what how would you talk to those vendors because they may say that's the way we You know one of the things that make open stack such a rich ecosystem is we allow people vendors the ability to Innovate and to add value Back from their solution and now it sounds like I know argue could be made that you're trying to take that away So I this this has been an ongoing debate right and this is the stuff that I've been pretty public about and I've written about so my perspective on this is Those taking the idea what Ken's talking about is taking the idea of hey Here's a driver for cinder that actually does all of this fancy scheduling all of this advanced control path stuff All of these things that we're talking about up here today It does all of that just plugs into cinder and you're done my argument against that is that actually is detrimental to the cinder project because the point that I'm trying to make sitting up here right now is all of that functionality is the functionality that we should be putting inside of cinder not in a cinder driver So the fact is is you know people can sit up here and say oh well It's really hard and it's watered down and you have to pick the lowest common denominator Well these drivers that they're proposing are doing the exact same thing So all I'm saying is there's absolutely no reason to actually have that extra layer The reality is we should all be focused on actually putting that functionality in those those features inside of cinder itself and making cinder better as opposed to Basically forking out all these different drivers Anyone else? Yeah, so I completely agree with that and I think you know that that kind of Broadens the conversation to it is cinder's role beyond just you know block storage as well as being forward because that's one of the key Alments that we've been talking about as well as as you know the SDS notion of block versus block object and file and Then on top of that, you know the layer cake if you will is definitely not Productive, but at the same time if there's no alternative you know It would be nice to have it as an option But the game plan long term should be to have a model where you don't have to have a layer cake necessarily Another box on the panel So my thoughts if you were at the line of summit know that it very much echoes what John has already said and my my main fear about it is that it's going to create silos inside of the storage project and In general, I just I feel like that You know once You know like what what is the point for a lot of us to even meet? About at for the cinder summit at that point about what new features that we want to introduce Because I mean at that point I kind of feel like well vendor a and vendor B is already doing I don't even want to think of the nightmare of what the that feature matrix is going to look like at that point in fact, that's very much what we got rid of and so if I have you know a Variety of vendors that could do something a lot better than what can be done inside of cinder and while the I fear at that point there's going to be this shift where there's going to be some of us still meeting at the summit and Trying to implement those features, you know they're still trying to work together in an open-source way and Unfortunately again, it goes back to there's going to be silos created and another feature matrix and it's going to be a huge nightmare for Operators to know what exactly They're going to get from this particular venture it's going to completely move away from the idea of what I feel like some of us have been saying on SDS and having this Again this headache that the operator is going to have to think about what they are deploying Between, you know two different storage two totally different similar or dissimilar storage systems. I Think it's um, I think it's a bit of a Identity issue also for open stack and cinder as a whole Because if you think about it every vendor who's involved in the open-sack ecosystem is incentivized to have as much Functionality in their particular specific proprietary plug-in as possible So they can sell their product and contribute the minimum amount to open-stack that allows open-stack users to leverage their functionality And I think our goal as a community and everybody in this room and people who are not Solely beholden to to their commercial entity is to actually build a larger community that that pushes as much of that development effort into creating common functionality that benefits everyone so that you can then spend the rest of your time or the vendor can Focus on making their product good and the community as a whole will benefit from all the common functionality And I think any in general if we sort of use that criteria for deciding what past to take for the project Then I think everyone will be a little better served Vendors are only interested in their driver and their plug-in Cinder in particular is is a good community where we actually have done a really good job of Vendors, even though we are very vendor centric and very vendor heavy They do an extremely good job of pushing their changes through all of the center code upstream and making center better And I think I'm example of that Mike is an example of that folks from EMC. There's a number of folks folks from net app How would you address customers ago? That's a great idea I love this idea of pushing software-defined storage into Cinder itself, but you the community moves too slow And I want the functionality. I need it now How about so I want to implement vendor a Software-defined storage solution. We but I also want Compatibility with the open stack. So I want to be in this part of the Cinder project How would you address anyone the panel? So Cinder today you can just the vendor can just supply their Customers with a Cinder driver right doesn't have to be merged into the code base in order to do that And it can comply with any version of the API. So there's no really no real issue there Right except some customer. I've talked to some customers who rightly or wrongly right? They see the the fact that the codes in the Isn't the repo as a kind of a stamp of certification like so that customers will say I'm not going to implement something Even those API compatible unless it's made it into the into the trunk So the way you work in open source is that you always when you when you work in a project You always take the the point of view of that project right and what is best for that project and in Cinder It's in Cinder's best interest not to allow these Drivers in because it forces the vendors to Provide this functionality in Cinder the same way that Cinder did revolutionized storage API's by providing a An API that all the vendors nowadays comply with right and it's open if we go to something like Viper from EMC Then that detracts value as John says and it drives customers to Integrate against a proprietary API so they don't have the freedom that they need Okay, so in general, I'd say that's the vendors problem, right? Okay Anyone else comments on that a box on how to help those customers who want the functionality Yeah, you know, so so this is an argument that that comes up, you know fairly often So I've been working on Cinder since it started I started the project and everything else And I talked to a lot of users a lot of customers and stuff like that The fact is I have yet to come across a customer or a user that says to me Hey, I don't have enough choices or enough options for what I can do with Cinder I have yet to have a single person come up and say that to me I have had plenty of vendors come to me and say that to push their Specific feature and stuff like that, but I don't have any real data So if there are people out here that are operators and users and stuff like that We would love that kind of feedback So if it's out there don't go to your vendor with it Please bring it to the open-stat community and let us know because right now. We're not aware that that's a big gap Okay, that's actually a good transition unless someone else has another comment is how do Obviously this panel is really just to be you know another beginning of this discussion. What's But I don't think it helps us to have this session only every six months All right, we need to have this as an ongoing discussion. So what's the best way? For example, you're saying hey, we want feedback from users and operators What's the best way to get that feedback to the Cinder community so that you can you guys can all take that into account? Mike Mike Ptl Well, you certainly know what I look like I'm kind of hard to not find But you know that won't scale So, I mean certainly you're more you're more than welcome to you know reach out to us I know maybe IRC and the open-stack mailing list may be a bit daunting, but That's currently how the open-stack community embraces ideas right now So I mean feel free to you know start up an idea You can also be suggesting Everything from the design summits I mean we take everybody's Different topics in and we kind of discuss them out whether or not this is something that we think can just be you know done Without a discussion or whether it needs an actual discussion If we don't think it would be make much sense over IRC so I guess what I'm saying embrace those You know different ways to you know get something out, but I also wanted to echo exactly what John said I mean I've also have been into plenty of speaking sessions myself and Going to a variety of conferences talking about open-stack and it's sort of the same idea that I don't think a lot of the users I talked to either you know, they're not sure what they want from sender yet Or you know, they're excited about what it does already and they haven't exactly dived into it yet But I've always been very welcome to anybody who wants to reach out to me about what you want to see next and sender I Just wanted to add real quick You know Mike talks about IRC and stuff like that The other thing is our Twitter handles are all up there tweet. Just send us a tweet. I can't even read it and I'm right here It's on so if you go to the schedule And look up the speaker profile there The Twitter's on there JDG underscore eight, so I mean it's we forward it to Mike Yeah, he's handing out my business cards this time so but Also, I mean I checked the open-stack user mailing lists You know every other day, so I mean please you know make sure to you know try to reach out on there And you know if it's something that somebody else hasn't already chimed on then, you know, I'll be on there So we got ten more about ten minutes So that's as I promised I wanted to leave the floor open for some questions from from you You know audience so if you've got some questions for the panel I invite you to walk up to one of the microphones and just speak out loud and try to trust them And if you have a question for a specific panelists, please state that otherwise, you know Anyone from the panel can try to answer Yeah, hi my name's Simon Dodson from pure storage just going back a little bit to what you were talking about You know the added features and deciding what's a standard or not a standard and what vendor provides what? Let's think about something very generic like replication How do you decide what goes into Cinder core and what doesn't and what has to be done via those drivers? Let's think because replication could be synchronous asynchronous full-copy based snapshot based Consistency groups no consistency groups. Where does that define cuz it goes to your point about the support matrix? So go ahead anyone? So the way we the way we handle that right now is is basically we just did Replication and the way we define replication is a use case And we say this is the use case that we're trying to achieve The problem like you mentioned is there's different types of replication It means different things to different people and it's implemented differently Where we're at today is that's an implementation detail The thing that's cool about that though is you can have a device that does Asynchronous versus another one that does synchronous and what we can do is we can set up things like hey Here's a volume type does specifically for Synchronous replication and here's one for asynchronous replication and then depending on which one of those volume types you're picking It's gonna choose the appropriate back end to get you what you want But does that net then not go to the point about the operators having to do that more much more thinking about what they've got in Their back end or not. I don't think you're ever gonna get to a point where the operator doesn't have to do any thinking right? I think I think that's something that is It's it's a cool idea and it's it's neat But the thing is is until you get to a point where you have a single homogenous environment I don't think that's realistic and I don't think it's what most people want most people actually they want to have diversity They have reasons for diversity right There's there's plenty of good reasons why you might need different types of storage and in different types of behaviors Whether it be cost whether it be power whether it be performance whatever it might be So I don't I don't think that should be taken out of the equation I think what Mike was getting at more was more about I don't have to worry about hey Is this API call gonna actually work? That's the kind of thing that I think he was getting at more as opposed to is it gonna behave exactly the same way? And what do I have to know about it? So you're talking about Effectively extra specs and adding those into the core Yeah, so right now we we do that through extra specs and it's a manual process and what I'm suggesting is in the In the future release what we should be doing is looking at things like how to make the scheduler more intelligent So how do we expose things to the admin to the ops guy that are ops girl that says hey These are the capabilities that are being reported from your pool of storage devices, right? And here's how you can set up different scheduling and stuff like that, right? So help you along with that and give you those tools to actually make it better So by putting those into the core for example, you would be standardizing So you don't go off and have a different extra specs that does exactly the same thing with a slightly different name from two different vendors Yeah, and that's the hard part right is is getting everybody to actually agree on what that is But I think if you draw a line in the sand and you start you can still do your customization and get things your own unique way through an extra Spec, but you can also get some goodness through the automation through the scheduler Thank you anyone else Just one more little thing so Of course the operator needs to know About storage but once you've defined Preset of policies that you want in your data center from that point on everything is automatic, right? You just choose what policy you'd like and it's automatically behind the scenes goes and provisions what you need Anyone else have questions Feel free to walk up to the microphone I was nice as you have any closing comments, or do you have questions for the other panels? I just know I just had a quick question because we were talking earlier about IRC and mailing lists So just quick how many of you are on the mailing list IRC already just to see how much feedback? Okay? Yeah, I was just gauging like how how many people are there so you can continue getting feedback. Hopefully Okay, anyone if anyone else Have questions you guys on the panel of any anything you want to close it And maybe a general question would be what? What kinds of features or capabilities people actually are looking to see out of Cinder? Or a control plane in general Well my legs asleep. Oh Hey guys Lee Calcoat from Cisco on two questions Going back to your prior topic on Vendor plugins and kind of pros versus cons on having you know sort of a ubiquitous amount of You know a common layer of functionality versus vendor specific capabilities. Is there It's a different way of phrasing that question. Is there You other than aspiring to have that functionality in Cinder is there at any point you guys would prevent or discourage vendors from building out functionality with any given plug-in Or do you just aspire to absorb it as you go? Yeah, I guess phrase differently. So Some of the vendors are contributing and building out their own plugins for capabilities and as you're going along your You're incorporating what you can into the into the community and encouraging them encouraging them to do so and I guess You know as it's sort of a sensitive topic from what I gather and this is actually my first time kind of paying attention to Cinder So apologies for the ignorance here, but is there a Had there been points in time where specific vendors have been discouraged from building out You know capabilities that that aren't contributed back or so it so if you mean building it out in terms of putting those capabilities inside of the driver and the driver only No, I don't I think there's certain cases where that has been true If it did things like change the behavior of the API Right, but in terms of things like, you know The the solid fire driver that I work on in particular is a really good example So a long time ago I put QoS support quality of service support in there We're the only ones at the time that had it. We're the only ones that do it the way we do it So it is very much custom, but the thing is is that's what we give the vehicle of things like extra specs And now we have QoS specs right because now there's other definitions of QoS But we provide that vehicle and we welcome that I mean, that's a good thing because because really the bottom line is is if you just give somebody one vanilla Thing for cinder it's not doing anybody any good, right? I mean, that's that's not making anybody happy So there has to be a way to expose those features and it is welcomed the thing that changes is when we get to this point where you start to duplicate features That are already in cinder or meet the definition of cinder or that should be in So makes sense to me. Yeah, it does. Um, last one is just a request for comment So so today Seagate announced their kinetic drive. I didn't know if you guys had comment on You know, that was a good thing or an exciting thing They've announced it at several different summits New and improved I think it's really nice to see people innovating at the interface level on the discs I think my only challenge to them is that they've sort of picked a new API that has a lot of Interesting properties and benefits or whatever but they still sort of fix the API and who knows whether that's going to be sort of The right way to talk to a disc so I'm Personally I'm more interested in the other other vendor HGST has a similar disc where you actually can put your own code on The disc and you can define whatever API you want Which I think is sort of opening up a whole world of possibilities. So I hope I hope that's to get Seagate will fall suit Thank you. All right, so it's I think it's top of the house, which means we're ending on time one quick thing is Mike what's uh You put a folks who may not know but what might be interested in sitting in some of the design sessions When is that and do you happen to know where we're aware? Yeah, so believe on on Wednesday We're set so We're over what is it what is it the hotel the meridian? Hotel and so just come on over we start at 9 a.m. So bright and early and Yeah, just find me as that's a good time Also, if you if you see anything I've tried out sender have anything missing You know reach out to me and as well as the rest of the core team They're sitting up front too of sender. So yeah and John, of course Right. Thanks, Mike. Okay. So thank you palace Thank the audience for joining you