 OK, good morning everyone. So I'm Antoine Cabot. I'm a cloud computing lab manager at a private research institute based in France. And today we talk about Watcher, a resource optimizer for OpenStack, plans for Newton and beyond. So first of all, I will introduce you to the Watcher mission and the key contributors we have in this project for 18 months now. And then Joe Cropper from IBM will introduce in more details about the architecture of Watcher and the other OpenStack components we are using in this project. And finally, Susan Bale from Intel will talk about what we did during the previous cycles, so MetaCast cycle specifically. And then the plans we have, the plan of actions we have for the next release for Newton. Of course, if you have any questions, feel free to ask them at the end. Please use the mic. So about Watcher mission, so as I said before, Watcher is a resource optimization service for OpenStack. So as I presented it a lot of times during this summit, many people told me that there are many different ways to do resource optimization. And that's right. It really depends on the kind of cloud you are running. So it could be public, private cloud. It could be you could use small VM, big VM containers, small flavors, big flavors. So every kind of cloud are different. So building an optimization service on top of this is pretty hard. So what we decided to provide with Watcher is a framework. So this is very important to understand that we don't provide any magic solution to optimize your cloud and reduce your TCO. We just provide a framework where you can build your own strategy, adapt it to your own environment. So the idea is really to provide all the base code that we'll be able to handle a strategy and then run it on top of your infrastructure. So in this way, we provide a pluggable architecture that allows you to provide strategy. So of course, we provide basic strategies like basic consolidation or energy aware optimizations like outlet temperature. So this is something we will talk in more details after. But the idea is it's really easy to build your own strategy, adapt it to your whole goals or features you want to have in your cloud. So this project, the project is out of the box value. So it means you can just run it on top of your infrastructure and then build your own strategy. Many people during this summit told me that, OK, that's cool. In fact, it's kind of like DRS for OpenStack. So I said, OK, so if you are familiar with the VMware environment, it could be like DRS. But the difference here is that we allows you to provide your own strategy, your own algorithm. So the idea is not to build a black box. The idea is to provide a way to build your own strategy of optimization. So some contributors we have today in this project. So we started this project 18 months ago. And now we have IBM become an Intel as the main contributors on this project. And we also have contributions from University Lab in Zurich. And AT&T Walmart has deep interest into this project. Also ZTE, Orange, and also a team in Russia. So we are lucky enough to have a team all over the world. And the watcher code is evolving all the time, 24, 24. So yeah, that's great. Of course, during the next cycle, the Newton cycle, we hope to have many more contributors on this project. And we plan to be under the big tent during this cycle. Everybody in the room, hopefully, will be there. Their names and companies will be up here next release. Yeah. Actually, I should also say that we have currently some blueprints that needs new contributors. So we are really welcoming people coming to this project. So coming back to the key features we provide in Watcher. So Watcher provides cloud optimization, for example, of using live migration in case of imbalance detection. So we will look at all the CPU metrics we can get from the telemetry system and then take decision about migrating one VM from one or to another in case of balancing the download. So as I said previously, it's a very flexible plugin structure. So as we can discuss later, you can add strategy. You can also add actions. You can add goals in Watcher. So everything is, could say, plug and play. And we provide under shelf optimization strategy today based on CPU RAM and energy consumption. So we have today three mode in mind. So the advice mode is the default mode to run Watcher. So you will install Watcher and then Watcher will run an audit on your infrastructure. So you just ask him to run an audit. And then after the audit, you will get an action plan, so a plan of actions to be run on your infrastructure to be optimized. So this is the advice mode. So the administrator can really say, OK, here is the audit results. That's good. But I don't want to run it because there is something wrong with this. So I don't want to run it. So you can really choose what you want. And there is an active mode for always on optimization. So the idea here is to provide a way to constantly optimize your infrastructure by running audits every 10 minutes if you want, and then run all the optimization automatically. And finally, there is a verbose mode for detailed optimization decision tracking. So we will have you to run step by step the optimization. So Joe will go into more details about the Watcher inside the OpenStack ecosystem. Yeah, thanks, Antoine. So we wanted to show up a picture here that talks about how does Watcher interface with the rest of the OpenStack components. Of course, there's a lot of different things we can do with optimization. And there's a lot of projects out in OpenStack that do a lot of really great things. And so rather than we're not trying to reinvent the wheel, in those cases, we're trying to leverage the services that are provided by the other projects to essentially optimize the data center. So if you think of this as sort of a hub and spoke type model, and this is just a very brief example of some of the things we can do. So I mean, Watcher can sort of sit in the middle. And then for Antoine was mentioning, you can do VM live migration. So we just leverage the services that are provided by Nova to invoke VM live migration. Of course, we use Keystone for all of the authentication, Oslo for all the common libraries. In terms of getting the data, Watcher isn't responsible for collecting all these metrics. There's other projects such as Solometer, Menosca, and such that can help us with that. So we're trying to leverage the value that all of the other projects in the ecosystem provide ultimately to deliver Watcher. So again, and I think that's one of the cool things with OpenStack is more projects are introduced and they provide a lot of value. Watcher can then continue to leverage those and plug in. So again, not trying to reinvent the wheel, we're trying to leverage the services that are provided by the other projects. And again, this can really enable new ways to get additional TCO and reduce the TCO for your data centers. So this picture here kind of shows an overview of the overall Watcher workflow. And I have to admit, so there's a lot of things that are on this picture that are very theoretical. Some things we've actually worked on implementing. There's some things that we need to do in the future. And they're sort of color coded as such. So the things that are green, we've spent a lot of time in those areas. Things that are orange, we've done a little bit of work. And things that are red, we need to put a lot more work in there. And frankly, that's what we're going to be focusing on for some of the upcoming cycles. So if you think of this as a continuous optimization loop that's continuously evaluated, and it sort of starts at the bottom with the monitor. So this is all about Solometer where we're actually trying to go ahead and collect data, collect metrics, what type of CPU utilization, memory utilization, IOPs, trying to get all this information from various data sources. Then that information can be analyzed and aggregated in a way that can then be used. We have a profiler component. Again, the idea would be that you could eventually predict and determine what might happen in the future after seeing patterns and whatnot. And then you have the actual optimizer, the thing that's actually going to work on trying to put together your optimization plan. And as you see an input to there, there's different objectives and constraints. So as an example, Congress could have different constraints that could eventually feed into this or the Nova scheduler of people have created things like co-location rules, anti-affinity and affinity. All these things can be input. And when Watcher tries to make scheduling decisions and make plans, we can leverage some of these other constraints that other projects have defined in the ecosystem. Then we go from the optimized step into the plan. So the planning is ultimately where Watcher will decide that virtual machine A needs to be migrated from host one to host two. It puts together a set of steps that can be acted upon. And then they're all ordered. And some steps could perhaps happen in parallel. Some things need to run serially. And then ultimately you have the executor, the thing, the supplier that goes in and runs all these steps. And this again is a continual loop. This isn't, sometimes when we talk to people, sometimes they have this notion of that staples button, right? You press the button, okay, that everything, that was easy, it was perfect. That's not the case, right? I mean, the clouds are very dynamic in nature. So things are never gonna be perfect. And you need to continually evaluate this in a loop to kind of, the idea is that after this loop runs, things were better than they were before. But you're gonna need to continue to do this over and over and over again. So this picture here shows a little bit of an architectural overview. So again, that last one was a little bit more theoretical in what we wanna do. This actually showcases a lot of the components that exist today in Watcher and sort of how some of the things exist. So if you look in the upper left-hand corner, and by the way, this model tries to adopt very similar architectures to that of a lot of the other OpenStack projects, right? So we have, there's Watcher CLIs. We have a plug-in for Horizon. And basically that's where the administrator or the engine can automatically invoke the various Watcher APIs. That comes in through a Watcher API so it's all rest just like every other OpenStack service. There's the OpenStack bus, so all the different Watcher services all communicate with each other over the message queue. Again, just like everything else in OpenStack. We have the Watcher database, which is MariaDB-based. And then if you look in the lower left-hand corner, you have the components where we're actually gathering metrics. So these are again, Solometer. We have integration with Solometer today. Things like Minoska will come in the future. So any hard lines you see here, those are integrations we have today. The dotted lines are things that we would like to do in the future. And then again, all the Solometer data goes into, we basically have time series databases, which is for the reference implementation. We use influxDB for that. And then you have the, sort of in the center there, you have the Applier, the Planner, and the Optimizer. And really the Planner and the Applier are sort of a logical subset of the overall Optimizer. And again, the Planner is the component that actually again takes all of the optimization data, the metric data, puts together that plan and then it's executed. And if you look in the upper right-hand corner, we currently have interaction talking with NOVA. Again, to do VM live migrations. Eventually there's a lot of storage-based optimizations. We can do networks. We want to have integration with Neutron and Cinder and things moving forward. So this next chart here kind of talks about the overall workflow. What are some of the objects or entities you deal with within Watcher? And I'll kind of go through this and there's the subsequent slide actually has a better picture of this. But basically the first thing you do, Antoine was sort of talking about in his initial discussion that you really have different strategies and goals that you want to achieve. And we provide with our reference implementation that there's a few out-of-box things that can give people value. But one of the real benefits of Watcher is that it's very extensible. So you can go in and add different plans or strategy that's gonna help you in what you're trying to achieve with your clouds or data centers. So you create a goal and then you create an audit template that gets connected to this goal. And the audit template is sort of a one-time creation. You create one of these. And then in step three, the administrator, or you could have a service that does this in an automated fashion, goes ahead and actually creates an audit, which then uses the audit template. So just as an example, a perfect strategy, an example of a strategy might be I want to stripe all the virtual machines in my cloud or I wanna pack, I wanna fill up one host, move on to the next one. So those are examples and that can be some really simple examples of a type of a goal you would create. And then the audit creates all of the plans or all of the different steps that need to be generated. And then the applyer again goes in and apply those. And so if you look at this in a picture form, I think this does a nice job. And by the way, we'll have these charts up so people can come back and also reference these as well. But I think if you look over at the administrator, the administrator triggers one of these audits. And again, if you look in the upper right-hand corner, again, that's where you have that goal. So that's the one-time configuration that again, we provide a few out-of-box things, but then people, contributors can add additional goals and strategies to whatever your heart's desire is. So the administrator triggers one of these audits and then based on that audit, you can do that from the CLI again, or you could have some cron-like job or having it happen in an automated fashion over and over and over and it generates that plan again. And then the plan is a series of steps that can be executed and that's all using and modifying the resources and you're called moving VMs, resizing VMs, maybe eventually moving volumes around, things of that nature. So I guess what I wanna do now is turn it over to Suzanne from Intel and she'll talk a little bit about some of the outlet strategies that Intel helped develop here and then talk a little bit about some of what we accomplished in Metaka and our plans moving forward into Newton. I'm sorry, thank you, Joe. So as Joe said, my name is Suzanne, I'm with Intel and some of the things that we contributed to watch or besides just hardening the infrastructure are some of the algorithms that we've been working on with customers around data center optimization. And so one of the examples that we've had is this outlet temperature where some of our customers are trying to raise the temperature in the data center, look at the spec for the server, everything looks fine but then some of the component behind the servers might start melting because the temperature gets too high and so just as a quick example, given that this strategy is pretty simple, we just wanted to go over what are the things that we actually doing in this strategy. So in this case, the outlet temperature require a server that has IPMI and the power thermal on the kind of Intel power thermal and given that it has those features, you can actually read those telemetries out of a cylinder and then start applying this strategy and this was just kind of a quick examples where we start up by looking at, does this hypervisor have outlet temperature? If it does, then we filter those and we look at, okay, is the outlet temperature for that hypervisor over the threshold? So it's a kind of a simple algorithm in this case where we look at the temperature, is it over the threshold? If it is, yes, then we can take an action. If it's not, we just leave it alone. In the case where the temperature is over the threshold, then we have to pick a new VM and so we use the constraints that we were talking about that are already in the system to pick a new VM and then given, once we do that, then we eventually end up with the action of doing the live migration of the VM. One of the important things here also, and you'll see this in some of the blueprint that we have for the Newton releases, it's great to be able to do this, but it's nice to actually understand what is the administrator going to get out of this? And so we have a blueprint around efficiency indicator where the plan is to come back with an efficient indicator to the admin before the admin actually trigger this action so that the admin knows ahead of time what the benefit is going to be for this move. So if we look at this slide kind of tells a little of the story of Wattra and where we're heading. So as you can see, the project was originally started in November of 14 by, I guess, Becom, and then we met in September of 15 at a kind of a meetup and that's really when the project kind of got kicked off. We've had, we're now following all the open stack processes and have been working really hard during the Mitaka release to behave like an open stack project. And we've been looking at several strategies during the Mitaka release. One of the things that we wanna do as soon as we actually come back from this summit is to start applying to get under the big tent. I think it's very important for us to be under the big tent and we are ready. We've had several talks with several of the TC members and they seem to be agree that we are ready. And so as you can see here, that should happen during the Newton release. So in terms of what did we do during the Mitaka release? So our kind of goals or focus were really two-fold. We wanted to improve the framework of the water project as well as look into creating additional strategies. And the reason we wanted to make sure we have strategies along with a better framework was having a great framework without strategies isn't very useful. But now we have, we're starting a small library of strategies that you guys can try out and you can actually customize them. And the idea is really here that this framework is totally pluggable. So if you decide to create your own strategies and even if you don't want to open source them, there is, you know, you can still use them. So going back to what we did during the Mitaka release, we really, again, focused on making sure that we had all our processes set up so that we are a real open stack project. So we did the salameter integration, DevStack, we have a DevStack plugin which means that if you want to try to take it for a spin, it installs very nicely in DevStack and you can try and try a whole bunch of things. The one thing I think we're the most proud of on this list is really the multi-node support on the gate because one of the things that we've done and I think very few projects have this at this point is that as you submit your patch as part of the testing, we're setting up a DevStack with two compute node and are doing live migration as we are testing. So we're really happy about, you know, being at, we're actually doing real testing of the features and not just the tempest test or some of the, you know, inline testing. And then the other things we've been working on is really making it easier to, for example, have parameters like threshold optimization passed into the strategies and of course also the watcher dashboard. So as I was mentioning earlier, the kind of second focus for the Mitaka release was really creating some strategies that people who wanted to try out watcher could, you know, so that they didn't have to make up their own to start out with. So at this point, we have the outlet temperature strategy based on the thermal telemetry. We also actually using other thermal telemetries and putting those in salameter for future algorithms around risk minimization of different components and also some of the other kind of thermal algorithm that we wanna put in, we being into the open stack. We're looking at, we have several kind of algorithm around rebalancing of VMs. There, it's both, you know, based on usage. We have one around an algorithm for a standard deviation strategy. And I think the important thing here is, so we have three algorithm around rebalancing and that is really because we've had three company needing different, having different needs and wanting to do different things. And so at this point, everybody has decided, you know, let's contribute them to open, to a watcher. But we could imagine, and we, you know, we could imagine that as this library grow, maybe we'll have like, you know, a watcher library of algorithm or where you can just pull them down from there so that they're not part of the watcher itself. But again, and like I said, we are, I'm sure, you know, we also are going to get the customers or companies that do not wanna open source their algorithm and that's totally fine. So what are our plans for the Newton release? So one of the things that we see a need for is really being able to integrate with what I would call external machine learning tools and frameworks. So in the case of Intel, we've opened source something called the trusted analytics platform where data scientists can sit and create the models and actually run them continuously and train them. And so one of the things we thought is, hey, wouldn't it be great if we took those models and actually had them applied to watcher? And so we're currently working on a scoring module that would allow that to happen so that watcher strategy could actually be based on an external model coming from like, or an external scoring engine coming from, in this case, it could be TAP or any other external machine learning tool. So we're kind of looking at that integration as well. One of our important blueprints, this release is also going to be the active mode so that we always have the optimization is always running. And we can use that for, again, anomaly detection, but also running your algorithm outlet temperature or fan speed or anything was in your infrastructure. The other thing is, we're looking at the verbose mode for detailed optimization so that you can actually debug this in a little better way. I've already talked to you around the efficiency indicator that would actually tell the admin that as part of the action plan, we would return this efficiency indicator and then we would return this efficiency indicator so that the admin or the operator would know ahead of time what kind of improvement can I expect out of my cloud and so that is something that we're looking into as well. One of the things that we also want to do is, right now, an action or an action plan is kind of for your whole cloud. One of the things we want to be able to do is actually create an action plan per entity or per group. So for example, if you have host aggregates, we want to be able to say create, optimize just for this host aggregate or for this group and not across my whole cloud. And of course, and this is where we want to invite kind of everybody here in the room is to provide more added values, optimization strategies. I think the strategies are really some of the things we're having talked with a lot of you where a lot of companies are thinking that that's the added value of this project is being able to come up with their own strategy, sorry, strategies that can then optimize their cloud. And so, you know, this is really where we're hoping that if you're having, you know, that where you can start contribute and also collaborate with us on the on watcher. So if you want to learn more, we have a wiki, we have OpenStack Watcher IRC, we meeting every week. And so, you know, let us know, come up and talk to us and we hope you want to collaborate with us on this. Thank you. And although we're up here, there's some people from the Watcher dev team I see out there and a few people back there. So again, thanks everybody. Appreciate your help and everything through this release. So, questions? Is this on? Can you hear me? We're Broadcom, we have a data source that is almost analogous to like heat sensors and stuff. It's real low level with network switches. How would we contribute to this effort? Because obviously we might have data that helps in an optimization scenario, right? So, yeah, I mean your question is really, so you have different metrics about different compute nodes and things and how can you feed that into the system? Yeah, we have an upstream project that collects like really low level stuff off of our A6 and silicon and pushes it out and puts it into Manasca. So, is it just simply publishing it to Manasca and you discover us or are you looking for, is there something else we would want to do to collaborate with you? I mean, we could talk, so as if you remember back to, I think, let's see if we go back to, let's see which chart was it. So, this slide here. So, right now we're integrating with Solometer and can pull metrics from there. We don't have integration with Manasca yet, but that was something we were looking at doing. So, once that line turns from dash to solid, we could look at where your information, where it's stored in there. And then, I mean, ideally that would be, all you'd have to do is get it in there and once that piece was done, then that information could be made available and could be used for strategy. Maybe find you on IRC or something and say we're publishing this stuff. Yeah, yeah, we could chat on IRC and actually kind of dig into that. But that's kind of the cool thing about this is that people are pushing their data into these other sources and we give you out-of-box integration with that. Then it makes people that actually want to go ahead and build strategies and whatnot around using that data. It makes them pretty seamless. Ed, one other question. Earlier talk, I learned about Congress. Is Congress on your radar? Is it a project? Yeah, Antoine, have you talked to them on this? Actually, we had a quick discussion with Congress. I saw the team is here. I think if you just come back to the loop here, we have something called the constraint on top. So we know that at one point, we should probably exchange with Congress to get all the SLA thing, I mean the constraints that the administrator has put on the infrastructure. We need to get them and then put them into the decision engine. So that's possibly another path for my data. I published a Manusca. Congress has some kind of policy that they're monitoring. They see that some microburst has happened in a switch. They end up pushing something back into Manusca that ultimately you would notice in that original pipeline where we're discussing because it's maybe an alarm or an event that's like screaming, there's a problem or possibly Congress is feeding into the optimized bubble itself, right? I think this is a use case I discovered just two days ago. So I think, yeah, there is something here. Yeah, I discovered Congress this morning. Yeah, I think both of those could be true. I mean, just as an example, one of the constraints we deal with today is things like if people are familiar with instance groups within NOVA. Again, that's where I was talking about the affinity and anti-affinity when watcher is actually trying to figure out where to move a VM to, we delegate to the NOVA scheduler. And so that kind of implicitly feeds in constraints. And so eventually we could have some sort of integration there with things like Congress and whatnot to help make decisions that honor everything you've set up in the cloud. Well anyway, we're interested in playing with this and talk to you guys offline about it. Yeah, yeah, come with us down in IRC, that'd be great. Let's just go back to this. Thank you guys for your talk. I'm just curious, is there much difference between optimizing VMs and containers? Is there any challenges with that? Well, yes, I mean, yeah, watcher can be very flexible. Right now we're actually focusing on optimizing, we're looking at really the compute nodes and then focusing on a lot of the virtual machine integration, so being able, I mean, for example, being able to live migrate and resize and I don't know, I mean, maybe somebody knows, I mean, last time when I was originally reading, I know like some of the containers weren't migratable, maybe that's changed now, but you need to be able to shift resources around and do some of that. We've talked about doing optimizations in a container space, but right now we've kind of been focused on the virtual machines, but I think a lot of the same use cases could apply there, but that's where we've been spending our effort. And we'd love to talk to you if you have a specific use case. So I'll write up on the concept of a rebalancing containers and based on what I can understand, you can, it sounds like you just kill them and set them back up or either that or you don't migrate them. And so if you have any use cases where you would actually migrate or move containers and how you would do it, come and talk to us and we'll be happy to actually add that to our use cases and understand how that would fit into the overall architecture. Thanks. Hi, thanks, very interesting presentation. Susan, I think you mentioned like you were interested in the possibility of applying machine learning to Watcher. Yes. But I mean, if I understand correctly, it seems like some of the monitoring functions would be done outside of Watcher and then you would use Watcher more to make the policy decisions or make some actions based on kind of distilled data that's coming back from the monitoring. So wouldn't the monitoring more naturally fit with the machine learning stuff fit more naturally with the monitoring function rather than directly at Watcher level? So, I guess it depends. So the external, I was talking about the external system. So the external system like the trusted analytics also has access to the data. And actually we were just talking about using that whatever the scoring engine coming out of those in Watcher so you can kind of look at them as like look at Watcher as a tap as publishing the scores into Watcher and Watcher using those to make its decisions as part of the strategy. I mean, ultimately it would be nice to know, right? I mean, a lot of the way Watcher works now is more retroactively, right? And it would be nice to get to a model where it can be proactive, right? I mean, that's really the future use case is to be able to make changes in the data center before the problem actually happens versus waiting until it hits that problematic point and then doing things. So that's kind of why we were trying to get that integration there is to avoid that problem before it happens. But it just seems like you'd want to put it as close to the data as possible. I don't know. Yeah, that's a fair point too, yeah. All right, thanks. Thank you. Oh, I was very excited to learn about Watcher. I'm with Serbo actually specializing placement analytics in virtual cloud environments and mostly dealing with VMware environments where JR Ass is there. So we're kind of looking, is there an equivalent component in OpenStack? So my specific question is, you showed the outlet temperature plan. And I'm wondering, well, how does that work with potentially all the other constraints you might have? Like when you pick a VM to move, how do you know that VM is actually movable? And when you pick a hypervisor for it to go to, how do you know Nova scheduler might not have a rule against moving it there? So this was a very simplified version of the algorithm. If you go and look in the algorithm, you'll see that we actually ask Nova for what nodes are available and what nodes are actually, so we make sure that we don't violate any of the constraints. So we're not just moving stuff willy-nilly, we're actually making sure that we are not breaking any of the constraints that are already set on the VMs. Excellent, thanks. And then follow up questions too. So once recommendation is made and there is an action plan, if that for whatever reason fails, is there some kind of feedback loop to that saying, okay, I'm gonna try something else? Yeah, so, not yet. So this is something we are working on, the applier, it was in red, so this is something we need more work on this because today we ran the action plan and if something failed, we just stopped the action plan. So it's very basic and we are looking at rollback actions and so on. So this is in our roadmap, but yeah, it's not available today. Excellent, thanks a lot. And I think though just to add to that, I mean, you start running this plan if it fails and that's kind of what I talked about. It's a cloud, so things are very dynamic and so the idea is okay, eventually you get a new plan and it can run and continue to try to make things better and we just hope that it's always, even if it's a little bit better, the state of it is a little bit better than it was before, we're happy and then you just kind of keep running some of those things too, so. Great question. Okay, anything else? Please go to the microphone. There's some kind of wondering where the boundary is between using AODH in open stack and triggering alarms based on some conditions like high temperature or some cash starvation and then an action plan for that. So that could be achieved with AODH and Congress. So where is that division of labor between Watcher and Congress AODH type of stuff? And unless we're learning the algorithms for the machine learning, I mean, it could be kind of accomplished with the alarm business and the Congress business, see? I mean, a lot of those are also based on, like you said, they're generating, okay, an alarm, something got exceeded some threshold, but then what? I don't know, I mean, you have to kind of figure out, what, I mean, where Watcher comes in is then trying to take action. I don't know, does that actually take actions based on that yet at this point or is it just? Yeah. I'm sorry, yeah, from what I've learned this morning that you said a set of conditions, it's somewhat algorithmic based on data that's coming in and then you can attribute an action with it. So, yeah, it sounds like there might be some significant overlaps. Yeah, I mean, I know we looked at that at one point and I mean, is that, is a lot of those set at say a host level or is it more an alarm? There's Congress people here, you said? Is it a VM? Well, maybe they could speak better than I. Yeah, there were some things when we started looking at where we were trying to find some differences in that mission and you read that, but I mean, that's a fair point and we could actually try to, you know, try to hash some of those details out with those folks. Right. I mean, it's a very interesting space and that's the thing is that, you know, you know, at IBM, for example, we were looking at some solutions in this space and, you know, there was a lot of stuff out there and that's kind of what really interested us to try to just come together and I think get everybody talking to hopefully build something. So we have an open source and open stack based solution that can do this because at the end of the day, you know, it's something I think open stack can really use. I mean, we're obviously all of us with watching everything else are really interesting. I think we're in an era of metrics and analytics and actionable items. It's becoming quite obvious at the summit to me. Yes. Okay, well, I think are we about out of time? Thank you. Yeah, thank you. Appreciate everybody's time. Thank you.