 Thank you so much for coming. I know it's late in the day. From the looks on your faces, you guys look as tired as I am. So I don't know, hopefully you make it to the event tonight if you signed up to go to that. I am super excited about my panelists today, too, that I've spent some time with before talking with on my OpenStack OS podcast and a new one who I can't wait to hear more from. And so we wanted to do a panel that really just talks to users who've been doing OpenStack for a while. Because coming into the fray, there's a lot of options and ways to consume OpenStack. These are three folks who have spent a tremendous amount of time building and operating and deploying at scale for production workloads. So I'm gonna let you all introduce yourself, Subbu. Hi there. My name is Subbu Alamaraju. Been at eBay for the last four years. We have been early adopters of OpenStack from 2012. I think one of the largest deployments now in production for three years. Have the experience of failures and success with OpenStack. So we can talk about everything. And it's just in case you didn't notice, this is his fan section here in the front. These are the guys who make it work, sir. Good. Hello, my name is Pusman. I'm a political advocate for these systems. Subsevered by the telephone. We are working for independent business companies. We have various public services in the United States. And if you want to know a lot about the telephone, you can tell us about the website there. And if you'd like to know. My name is Edgar McGenna. I work for Word Day, almost a year and a half. Word Day has been investing in OpenStack for almost two years now. We have a production system. We're moving all the bare metal systems into the clouds, in boats, virtual machines and containers. My team is not here. So I don't have a huge crowd, but I'm gonna expect Niki to give me a, there you are. That's what I'm talking about. And I'm wearing two hats all the time, which I love it. It's a user on OpenStack. And also I'm still a core developer for Neutron. So you have some complaints about that. I weigh you at the end of the session. Target practice, right there, kids. So each of you have, you are doing very different things with OpenStack, most definitely. Edgar, I know your deployment is relatively newish. You've been building it for some time. You have been running for a while at scale. And then you're working on a combination of both kind of public cloud infrastructure and then private as well. Tell us a little bit more about what you're actually doing with OpenStack. And I'll start with you, Supriya. So we started with a multi-tenant model early on, where we support multiple types of workloads on the same cloud infrastructure. The promise that we had, and the principle we had was you run a cloud-initiated infrastructure for multiple business units so that you can get the most efficiencies, get consistent experience and feature parity across different types of workloads. That has been the promise. And we started with our own distribution from GitHub and we packaged and rolled it out. We built automation over a number of years. It's still ongoing because there are always rough edges to polish. I mean, it's been serving production traffic. It's been serving DevTest traffic for a number of years. The promise of cloud as a self-service API-driven consumption has succeeded to a large extent. We still have a lot more to do, a lot more to improve security, availability and things like that, but it has been a fun yet challenging ride with OpenStack. The last time I talked to you, I can't remember how many VMs you were running. So it was a crazy number of VMs. Are you able to share how many VMs you're running at? I think today at a point, we are at a point where it doesn't really matter because I think the reason I say it is that when it comes to, once you find a pattern that is repeatable and it can multiply the number of times and you can grow the number as long as there's a business driver for that, but also we also worry about efficiencies and consumption and things like that. So it's not about numbers game as far as we are concerned, it's more about getting the service to the customer. What are you doing with OpenStack? Okay. We have various installations, so there are a lot of devices as well. On the other hand, we are currently working on a project that is in a lot of area far away. There will be an private public and local site as well. And on the project platform side, we have an installation for software as a sort of problem. And the platform we know it is software as a sort of software-based personal and human-sized enterprises. And OpenStack we have time to do the self, our own distribution at OpenStack. And then we have multiple development platforms inside of OpenStack. We are now at the set of 15 players that are in the distribution. And Edgar? So we're gonna start saying what the work thing is. It's a SaaS company, so we're as a service, so we basically have applications for human resources and finance. So we manage everything for our customers in our clouds. A few years ago, the company decided that based on the number of users that actually were so happy or worthy, we wanted to move from using just bare metal deployments into virtualization. So they started the journey of putting their applications on virtual machines and they decided to use OpenStack as a cloud management system. So we built an initial private cloud. We found a lot of challenge. So we're gonna start that our applications are designed to be multi-tenant by itself. So one bare metal server actually providing information from multiple tenant, which basically when they share the same networking we need to have a way to secure that these two virtual machines in the same virtual networking cannot talk between each other. That's the best way to define a security group for some of you that are not that familiar with that networking concept. In our use case, we have very few number of tenants, but a very large number of security groups. Such point that actually we have per communication between two VMs up to 35, 38 rules. I'm talking about different communication channels between two VMs that cannot be enabled and some of them that will be enabled. So it's a very unique use case and also in terms of the security requirements it's very aggressive. We have SSL enabled in all the OpenStack endpoints and we also have SSL enabled in the communication from these processes to the MySQL database or whatever database they use and also to the Rabbitin queue. We're using Rabbitin queue. So we also enable SSL. You can imagine, we manage payroll information, all these confidential information from our customers. So we need to guarantee that everything is secured and we keep it doing it. And so that's probably why they hired you with your networking background, I'm guessing. Well, yes, one of the biggest problems in Neutron is actually there was a performance degradation when you increase the number of security groups. So we look for different alternatives and actually we found something that actually fits perfectly what we're looking for. So we start using another open source player which is called OpenContrail for all the networking part. And is it seamless to the end users of the cloud? Is it like a network administrator that sets up these security groups but it's invisible to the end user? So end users never know what the application is running on, right? It could be a bare metal, it could be a virtual machine, it could be a container. That's a little bit the difference between our private cloud or any typical public cloud, right? So basically they keep having just URLs where they say myapplication.wordy.com. So we have low balances that actually descend into the specific place in the data center that actually will fulfill their requirements and they keep having. The best way to describe it is you power on the light, you have light, you don't care where it's coming from. You just like it to have it always running all the time. If that is off and your fridge is going off and your food is going bad, you get really, really upset. That is what we're trying to avoid for our customers. And so certainly in all of your cases, security is a huge concern. I mean, I can imagine with the number of customers that you work with from Deutsche Telecom, there's a lot of data privacy concerns that are always kind of consistent in your region of the world. Obviously eBay does a ton of transactions, Workday handles HR data and personal data from customers. Was that a prohibitor from each of you wanting to use AWS? Not necessarily, it depends on user requirements. In some cases in big projects, we are asked to provide AWS or Azure or something else. We prefer to provide our own platforms, that's obvious, but it depends on user requirements. I think if I may take the question, I don't think security is a criteria to choose between public and private cloud because my understanding is that based on what my experience with eBay and looking at the industry, it is more about how much have you automated? If you have a whole, that lets some person or entity do something that is not supposed to do, and then that can open a lot of things. So it's your maturity level in terms of automation. That is, I think, what is important for security. You could be, if you take a look at the case of Experian, I'm sure they had ton of firewalls and security experts in the world, but they had a leak for two years that nobody could find out. I think, so my bet is that it's about automation, whether using public cloud, private cloud, that's not really the point. It's about automating and making consistent, standardized setup for your things, with no exceptions. It seems like a few years ago, there were some concerns about taking customer data and putting it out in a public cloud. Like it's a multi-tenant environment, noise neighbors, less control, black box, you're not sure where it's running, who you're sharing with, who has access to those kinds of things. I think we've come a long way from that though, in terms of public cloud utilization. I think so, it's like you have a, the tendency has been in the past, but you put a firewall at the front of the room, but inside everybody can see everything. So if you let one guy in, that guy can steal. At that mindset it's not changing. So instead you're worried about point-to-point trust and confidentiality of data. So for us, the answer to the first question is yes or not. The first part is yes, because we actually leverage a lot of Amazon web services for development and only development. We actually have tools who can emulate and simulate tenant information into the Amazon web services. That's the way our application developer uses to add, fix, et cetera. In production, tenant information never leaves our premises. It's always in our private cloud. We keep it very secure. We have two layers of security. We have up to three layers of bastions to actually get into the data center. In my role as an operations architect, I don't have access to tenant information at all. It's probably better that way, huh? Oh yeah, right. I will be leaking, I will be leaking. Talking about security, it's not only about security in general, it's about data privacy as well. So there are a lot of customers that prefer to have their data stored in German data centers with respect to German data privacy laws. And that's a common requirement that we have. It seems much more so with the recent rulings around some of the safe harbor stuff that was going on. It's a non-clear situation now. Yeah, it seems like that's gonna be an ongoing thing. When we talk to our counterparts in Europe, I mean, that's the number one thing is data sovereignty. Making sure that the data stays in the country never goes out, which at some point, it can be difficult, especially if you're using SaaS services and you're not sure where those endpoints are. So talk to me about kind of what you're doing above the stack. I know you are a big proponent of automation. You have your, I forgot what your acronym is that you used on your website, but you've got like an acronym that you use and it's a process that you have folks in your organization follow. I mean to think what the way we are approaching the about the, sorry. The way we are approaching about cloud is I think if we, our strategy pans out and if we do the right things in a year, if sometime in future, we wanna make the cloud disappear from customers. In the sense that because the way we approached cloud in the industry is opens that included is that you have these APIs that you give out to your customers and they put together whether they are admins or platform developers, they put together infrastructure, right applications. But I think oftentimes they are not able to make the applications resilient to failure. We are expecting them to do it. And in fact, if you look at OpenStack, OpenStack doesn't have all the building blocks to make apps resilient. AWS has somewhat more complete portfolio, others to a lesser extent. So I think that problem hasn't been working out well for in industry today. So we want to find out patterns where the resiliency is built into the infrastructure into the platform we provide to our customers. Whether it's through past layers or something else, Kubernetes, cluster management, those kind of things. I think to me that the next evolution of cloud in general, I think that's where we are betting a lot. OpenStack remains, it still provides a building blocks for that layer, but it is that layer that hides customers from infrastructure failures. That's what we see. No doubt that in traditional sort of VMware environments that resiliency happened at the infrastructure layer. Now with vMotion and heartbeat and other things, shared storage was a thing. Now it's like, you have to plan for failure, but how many people are actually doing that? Like how do you get your users to develop against failure when it's not automatic, especially if they've been developing on a VMware environment? So something that I can share about that is we created a project that we call it Gourmet. It's on the top of the stack. It's actually a project that helps the developers to, by a simple API, upload their application package into the cloud. They don't need to know about Glance. They don't need to know about Cinder or Netron or OpenStack at all. They just have a simple API to say, my app is ready, put it there and I want to use it. And it's full automated. So actually for the project that we're leading inside of the cloud engineering team at Word Day, it's called WPC, World of Private Clouds. With the Gourmet project, what we ended up having is between 300 or 400 VMs daily being created from a golden VM that it was deployed that morning by the Gourmet team automatically. All the applications developer jump in, validate all the changes that actually they created and we drop it at the end of the day and we do these sometimes events like two, three times in one day. So you were asking before about how many VMs runs per time could be like, I don't know, 500, 600. It doesn't matter anymore. It's the ability to be able to create and delete all the VMs dynamically. Everything keeps running all the time. We've seen certainly one of our customers tap joy. The reason why they did Hadoop on OpenStack is because they wanted that ability, provision, tear it down, provision, tear it down, provision, tear it down. Are you seeing any internal requests for big data platforms on OpenStack? Yes, we are looking at ways of consolidating infrastructure. Today, typically in most organizations, you have Hadoop plans from differently from the rest of the application deployments and we are looking for ways to consolidate. I think we haven't made all the choices yet, but I think that's something we are looking at. To be fair, we are not that far with respect to OpenStack and these services on OpenStack. We offer various services to our customers, but we are also VMware fabrics or factory. So we have developer paths, we have big data services, we have SAP, all this stuff. But on the other hand, we have a lot of customers that think conservative. They have applications developed over 20 years and there is no return on investment to completely rewrite this application for a cloud native approach, for example. And if you would deploy all these applications on OpenStack, the stability or resiliency might be lower than a traditional cloud system. So you have to migrate together with your customer the application into another model, step-by-step, for example. And then you could introduce all these new services or could use the developer paths and all this stuff. And it depends because there is no, this customers, this customer, it's a huge corporation like automotive from the automotive industry, for example, and you have to find the right department. We have one project together with a huge company and they are developing their end customer applications on a developer path on our platform, but they would never do this with production applications or core applications they are hosting on our vehicle, for example. So it's a step-by-step approach to bring the customer into the new world on OpenStack, for example. Which is interesting because I feel like you've been there for a while. Yes, I actually want to challenge what you also mentioned before is that are we moving away from infrastructure being resilient to applications here being resilient? I think what has been happening if you look at it is that the rate of change in organization has been growing. The rate at which apps are getting changed, deployed every day has been growing consistently. Like years ago, eBay was doing bi-weekly, two weeks, there was a release day. Everybody sits down and then pushes code to the site. That's no longer the case. I can deploy now. I can deploy tonight in the breakfast. Anytime I can deploy. So when the rate of change is so high, things fail faster naturally because you're introducing regressions and you're touching both the control plane and data plane and that's where the traditional models of expecting infrastructure to remain solid are breaking apart because you're changing it so frequently. So you have to make the software take care of the failures more than in the past. I don't think it's about VM motion versus something else. It is the rate of change that is causing this problem. Like, our open stack cloud changes much more rapidly. Like the teams here, they are deploying code to open stack every day and our customers are deploying code on that on VMs and using block storage, object storage and so on. So it changes everywhere. I think that is what is happening now. Are you seeing that rate of change now that your workday cloud is up and running? I would like to break the protocol here so I'm gonna ask the question now. Just because Sue will say it's something very important. So I noticed that everybody, like when they're talking about performance and HA and all those crazy things, they just focus on the control plane. And if you think about the typical data center, when you get a phone call in the middle of the night, it's because the data plane is down. It's because the user who's trying to use the application is not able to reach. It's not about not creating a VM in the middle of the night. Who creates a VM, not a container in the middle of the night? It's about the traffic that you're sending to the data, right? Wow, that's, we know of it. I think he does. I think those guys right there do. So my point is like, you put a lot of effort in managing that you have a very good HM model because you're so worried about the control plane. What do you do about your data plane? How do you make sure that your data plane, it's up and running all the time with everything now is virtualized? So that's my question now. I think, yeah. I think this is actually an interesting debate. I think make it more exciting to have this. Yes, a debate. We get this end of the day. You need something to fight. I think what has, this is my gain. I want to contradict at this point here, is that I think with the way the cloud has been going on, data plane expectations are not as high as they were before. Our promise, for example, is that we take model of public clouds in eBay as well. We promise you that you're able to, there is enough capacity that you're able to provision, move traffic. That means these are all control changes. Like an operator is moving, failing over data center. That means he's touching a load balancer. He's config, he's changing its configuration. So the configuration, the need for high HA config changes has increased because there is an expectation that things are failing. We need to change quickly. You've given talks on the subject of HHA. What do you think, Gurd? That's a good question. I don't know. I don't know if it's really changing the expectation on the data plane. Are you finding that your application users are, is it become more, obviously you guys have been doing this for a while, so is it automatic now that they just need to be able to automate around failure? Or is there, are you making it easier by putting in some kind of layers, as Edgar's team does, where it's handling that HA for you? Like do you need to serve that to the developer or should you leave it up to the developer to handle it? Absolutely, it is some platform that is using the capabilities in the control plane. It's not the end developer, definitely, but you still need that flexibility. Like it is often our operations teams that are doing this failure. They need that knob. So the button needs to work. So in the case of the public clouds, obviously they need to be worried about the control plane, right? Because they actually open the APIs for their users. Their customers trying to create virtual decision. They tried to create virtual machines and just being the shoes of a customer, just trying to create a VM and put some stuff there, run in and then destroy it and blah, blah, blah. And you get frustrated because you try five times and out of those five times three of them didn't work. You say, this is not working. You don't even test the data planes or hookers or the data plane at that point. So the control plane doesn't work, you just throw it away, you go to the next vendor. When you control in the private clouds all the control plane, do you prepare your actually data center with all the VMs and the world flexibility that you want it? What you worry most is actually that your data plane is at the highest performance that you ever think about it. One of the major concerns of word data actually moving into virtualization is the performance degradation that we have moving in from the bare metal to the VM. We get between 15 to 20% performance degradation just because of that virtualization layer. While we test on containers we get less than 3% of performance degradation. So that's our way to go now. Nice. So containers, what technologies are you using currently? We don't have any production yet but we're testing Docker containers. Docker? Are you doing containers as well? This is another container talk. Yeah, I mean, yeah. Sure is. Yes, we are looking at it. I think again it's in the context of we're not looking at containers as replacement for virtual machines or bare metal but we are looking at containers as a way to better implement cluster management using Kubernetes. I think that is our driver. Containers you could have done last year but we were waiting on getting the right cluster management model already in the industry. Are you seeing any demand for containers yet? Yeah, I saw an announcement last week that on our internal build systems containers are available for the developers for testing. We don't have it in production. We are using Docker in this system. But there is development in this direction so I think in a few months we will have something that we could use on our platforms as well. It doesn't matter how awesome the containers could be. VMs are not going away, by the way. So they're gonna be there, they're gonna stay there. They need to co-exist some way. Even the bare metal, right? This is where we have an ironic because we found that actually we still need things running on bare metal a couple of weeks ago. Even if you did though, there wouldn't be enough people who knew how to deploy on them anyway, right? It's still such a new technology. I think there's still people who are trying to make the transition from bare metal just to cloud in general. Like, you know, what percentage is the technology change? What is the breakdown of moving to cloud between the technology change versus the cultural change within your organizations? I heard this from the director of infrastructure. If it works, don't touch it. You can imagine how difficult it's actually to make a change in the infrastructure systems. It's very scary. We have SLS with the customers and we have like, in what is like very crazy, we have per month, I think, it's just max of four hours of downtime and if we break those four hours, actually we start paying back to the customer something that's crazy as that. So it's very scary for the infra things to introduce any kind of changes and they have to be worried because these guys are the first line of defense in terms of something is not working and they really woke up in the middle of the night and they get into pajamas into the knock room and actually trying to fix things as soon as possible. That's true for us as well. The whole organization tried over decades to prevent failure and now one of the paradigms of cloud is, yeah, you will have failures but you have a massive amount of VMs, for example, to fight that, that's contradictory. So it's a problem. It's a huge cultural change and all the processes and the organizations and it's not that easy and every new technology that's coming up, for example, containers or whatever, cloud native applications, for example, it's again a cultural change that is needed. I think it is an absolutely necessary change because an younger organization that's coming up today has the advantage of an older organization that is monolithic and has a huge cultural mindset of no VMs or bad or VM at least great and it's actually going to admit, I think these companies will find it harder and harder to survive going forward. I think there's no... Because they can't keep up with the rate of change, right? Exactly, you die, you perish in this business, if not. And I just want to finish with something. I'll say that is what the infra guy was saying, it's not what I think about it. I think what we're building with OpenStack and all the SDN and all these fancy concept is to actually break that concept of not changing because you're afraid things are not going to work. What was before a router? It was a big piece of hardware that actually you buy with a very expensive price and money. You put it there, you put another one because it has to be a check and a think and you don't touch it for ages, right? What is now is an API, a new API that actually you call there and a simple thing that actually you configure with a couple of seconds and you actually could enable two virtual machines or two containers to actually to talk each other. So things are changing. I mean, that's the whole new world of virtual decisions. On the data center, you have the ability to draw a line between what is the physical world and the virtual world. You can actually now manage everything that you manage in weeks on the physical layer in seconds on the virtual layer and that's what we're trying to introduce to the infra teams. How are you, what are you doing to get the adoption rates up? Like, did you, at some point, I think you guys pretty much force people. Is that accurate? No, we didn't have- We got them wrong. We didn't have to, I think it is the automation that helped the adoption of CloudDB. I think once we put a platform where any developer can log in and deploy his or her app anytime that just drove the massive adoption because they didn't have to wait. That's awesome. So it's more appealing to the, I don't freaking wanna wait for resources, give me what I need right now. It's hard, you know, we talk to customers all the time and that's the challenge is, you know, they're like, oh, what about my ticketing systems? What about, you know, all these other things? How are you handling chargeback and billback? I mean, it's a buffet right now. It's a buffet, all you can eat. Sometimes we run out of food and just wait. What about you, Edgar? Are you guys doing billback chargeback yet? No, not yet. I think it's a good idea, I like that buffet thing. On the convincing part, I think it's about showing them how things could actually be better for their own life, right? How actually could actually improve their own processes. Forget about the old practices about, like, oh, there's a ticket and so we approve for ABC, security, BP, blah, blah, blah, blah, blah, blah, and then come back to you and then you do it. It's a speed up a lot of things and you can actually invest time in learning for new things. It's just kind of a little natural happening right now. Edgar, are you seeing the adoption rate of cloud really starting to take off? Yes, in general, yes. And what are customers wanting to do with it? You're talking about automotive customers, maybe some financial service customers where data sovereignty is a big deal in your part of the world. What are people wanting to do with cloud? In general, it's to reduce their costs. That's the first approach from my point of view. Yeah, it's cost reduction. And on the first step, it's an outsourcing model in many of the projects. But after the first step, when they adopted cloud technology, cloud processes, then they get a step further with new services from the cloud, outsourcing more services from the internal IT departments to the cloud and to the service providers. So it's a huge wave from my point of view. I find it's hard to put a price tag on agility. Like, how do you describe what the value is of being able to push code many, many times an hour versus pushing code once every two weeks, for example? Like, how do you? I think it is tough to put a number on that. I think, but it is clear that the business expects the agility followed by enforcement of security and availability policies. I think the business sees it clearly. I just have to put a value on that. It's not expected. It's about creating value. And I can steadily create value if I have the development process that allows me to do this constantly, daily, on daily basis. And then the traditional process I have to wait a few weeks, for example. And then I could introduce a new feature, but I have to again wait a few weeks to then discuss if it's implemented the right way. That's not very, very fast. And the business sees the advantage to create value steadily and fastly. Are you seeing a return yet at workday? Well, the company is based on that. When you just describe it, every Friday night we push a new patch into the whole data center. Every customer of what they use is the same version of the software, no matter what. We do patches every Friday night at 6 p.m. There is no downtime. The customers, they didn't even realize about it. And every thing is two months, we actually do a new release of the whole software. But things keep working the same way. So agility to actually do patches and to actually change the code, it's one of the most for changing the business model, right? In the past, you spend one year, two years, whatever, as a software group, as an application development team, you cook your application or your software and you ship it, you sell it, you pay a license for that for two years and then in two years you get a new one and then you spend all that. That's all business, that business is dying. Still there, but from my point of view it's dying. The new business is what most of these cloud companies is doing, right? You don't care about the software, you don't care about the customer's issues. It's up there, you just use it whenever you don't need it, you just drop it, stop paying for that, kind of like Netflix model, if you're familiar with that. Do any of you have any questions in the audience for our panelists here? Ooh, you have one. You have another one. Come on up and this is microphone work, yes? Test, test. Let me grab him and I'll bring it to you. Yeah, for Subbu. So how much portion of EVA is an open stack today? I can't give a specific number, but it is significant. Okay, so my follow-up questions are, can you comment on your issues with scalability and upgradability? I think we learned it by trial and error. We have not been able to hit the design limits of our deployments. For example, we designed for certain number in mind. We are not able to get to that number because we encountered issues along the way with various parts of open stack. I think, again, we are probably in the minority because it's not everybody has that requirement to hit that limit. I can share offline the numbers, but not in public of those scale limits. But upgradability, I think we had some marathon sessions initially, like upgrades used to be like nightly and we spend the whole night, hours and hours. Now we're at a point where we can do upgrades. We have upgraded decently HALA4AZs to, I think, ZUNO, and now we're gonna go to the next round of upgrades. They have become more seamless for two reasons. Open stack is getting better, and we're also getting better as an organization, maturing in terms of how to make changes to them. Are you getting to a point where you are hoping that the releases give you a little bit more time in between? Are you skipping releases? We are skipping releases, but I think we're also hoping to see the inner circle in the open stack, the six, seven projects, slow down and become more focused on stability and scalability and reduce the rate of change. Or do it the way that doesn't impact existing deployments? I have a question is, if your boss asks you to fix just one issue, the top issue, that only one issue that you can resolve from three different companies, what's your biggest pain and what you want to invest to resolve now? I think that's what a lot of people start at the cloud who will want to know. I gotta think a little bit. The biggest pain point for us, unfortunately, is upgrades. We got that big problem. We invest a lot of time on CI, CD. If you're interested, tomorrow morning, 9 a.m., talking about that, marketing. But yeah, upgrades is a big problem and it's not our fault. It's open stack fault because we keep changing, we keep changing the configuration files, the APIs versioning and all those kind of things, so what do we use, Chef, for configuration management? And our cookbooks needs to be updated every time that there is a new release. So we cannot do as much releases because there is a lot of uncompatibility. And then we change the schemas on the database and that's a lot of risk to actually trying just to push things on that. So Mamana will be in love with me even more if we fix that. Sounds like you need to find a good partner, you know? Yes, yes. I know, but Edgar gets a little baddie with the vendors harassing him all the time. I love them all. You love them all. Gern? Depends on the platform where we have the issue. So in one case, I would call the partner. That's the easiest way. On the open stack platform, that is kind of a do-it-yourself distribution, then we would have to do it by ourselves. In this team, we had developers and they had to fix it by themselves. Yeah, then they would do it. And on the other hand, we have support by the distribution. They would also help us to fix the problem and that's the way to go. And you're the last one because we're at time. So if I were to fix something, I would reduce choice in open stack. I don't mean remove drivers, but at the core of open stacks, there are some components that are too many ways of doing things, five, three databases. Like Neutron? Or messaging platforms. No, I think there is too much. I think some of the choice has moving parts and it's more complexity. I would simplify, reduce that. So it becomes easy for us operating cloud. Thank you so much. How about a round of applause for my awesome panelists? Thank you. You have a talk at 9 a.m. tomorrow, Edgar? Yes, yes. We're gonna share CICD and what is an awesome talk, actually. Edgar, do you have any talks coming up? Not this time? Subu? 9 a.m. on Keystone. Yes. Oh, that's gonna be a pie now. No, no, no. What? What? Thanks, y'all. You should say we are hiring too. Oh, they're all hiring, by the way. All of them.