 All right, let's get started. Thank you for coming. This has been a great show for Walmart. Normally, Walmart is not known for saying what we're doing publicly. So Amadeep was able to do a keynote. We talked as a group sort of on our journey. And today I'm going to talk about the lessons that we've learned from building an enterprise cloud. So one of the things from talking to people and talking to other operators is that finding out that a lot of us have similar things to say. And people aren't leaving enough time for Q&A. So I sort of have this talk split into three sections. And I'm sort of framing the problem of building a cloud inside an enterprise that way. What I'd like to do is have a lot more Q&A time. And after each one of the sections, people can just ask questions. Feel free to come up to the mic and ask a question or if you're buried deep in the crowd. Just yell out loud enough that I can rephrase it. And then we'll go from there. So we'll make this a lot more interactive than some of the other sessions I've been able to be. So it's great having sort of this nice intimate space here. Not too intimate though. A couple of things first, though. I guess I'll move to this slide, but a lot of people have said, well look, it's great that Walmart's running a cloud. I can't throw that many people at it. So let's dispel that. Our team's only 15 people. So we don't have a ton of people running our cloud, building our cloud and doing anything with the number of people that you'd think Walmart would throw at it. None of us have PhDs. So it doesn't take a PhD to deploy a cloud. It's not like it was in the Alamo days when you were trying to build it all from separate packages yourself and the descriptions were pages long for each project. There are installers and they're super good. And so it doesn't take an advanced degree to deploy it. Either that or there's a bunch of us in this room who have honorary degrees from some big vendor, right? The other thing from one of the keynotes that I'm gonna have to disagree with as well is that you don't have to have storage to have a successful cloud. If your workloads don't require block or object store, you can totally get away with ephemeral storage and we have. Now where I think they get the statement right is that if you do deploy that storage, you've gotta get that right because deploying crummy storage will probably kill your cloud. But you don't absolutely have to have it. You've probably heard most of what I have to say on this slide here. Pangea was the project that we took to rebuild the e-commerce platform. So I'm gonna skip a lot of the stuff that I normally would say. I think everybody knows who Walmart is and what we've done. We've had some great opportunities where I'm a deep and others have talked about. Some of our numbers. So I'm gonna just skip all of that stuff. I'm gonna jump into, these are sort of the three phases or pieces involved in doing something that really changes an enterprise. And this is kind of a standard way of phrasing some of the stuff. People, process and technology. And you really need all three in the sort of the intersection there to really change an enterprise. And if you're gonna cloudify an enterprise or run a cloud, you have to really change the culture. And those three pieces have to come together to let you do that. So you can sit down and rewrite applications like we did in Pangea. That's pretty easy. We know how to write code. In fact, there are thousands of people in this building who know how to write code. That's not hard. Not in the way that transforming an enterprise is hard. And transforming an enterprise is hard because people are hard. You've gotta deal with people. You've gotta change the people. And changing people is not an easy problem. I think we all know that. I like to joke that if you get two people in a room, you're gonna have miscommunication problems. Imagine doing that at an enterprise. So when we say, yeah, gotta add a Walmart, we have these miscommunication problems. Yeah, sure. But that's not a Walmart problem. I've had that problem at the smallest, smartest startups that I've ever worked at. So that's a human thing, right? You start out with a driver. You have some sort of driver that says, hey, we need cloud. We need to deploy cloud. That's what really lets you do something to change an enterprise this way. We started out, it started out as an engineering job. The engineers wanted some cloud. And that's where I actually started in Walmart. I started in an engineering group. You start running some capacity and pretty soon it becomes an operations problem. And now I'm in the operations group. And we have a much bigger team than just a couple of engineers running things, sustaining and iterating after this point. So I'll talk about some of that stuff as well. So the first section here is kind of the people problem, right? So, especially in a company as large as Walmart, you have a lot of people who have been there for a long time or who have done things a certain way. Most enterprises are gonna have this, right? You've got people who think a certain way, a standard, call it a standard enterprise way for good or for bad. But you run into the same mindsets everywhere, right? People at startups don't necessarily get cloud straight out from the get go. So you have to change their minds too. And that's one of the reasons that I love this quote here is that if you're gonna change the way we do e-commerce or you're gonna change to a cloud deployment or you're going to change that you're agile or you're using Kanban or you're really changing anything, you've really gotta change your thinking. And that was one of the ways that I framed coming to Walmart is that we were gonna use OpenStack to bring cloud to Walmart. But the real problem is that we have to change hearts and minds. If you don't change the way people are thinking and the way they're feeling about the technology and what they think about you're trying to do with to their applications or to their data center or to their clouds or to whatever else, it's an uphill battle that you'll never win. And that becomes a real hard clash. So talking to people and getting them to think the same way you're thinking and bringing them along into a cloud mentality is one of the hardest things to do. It certainly helps if you have some buddies in the organization who are thinking the same way as you or as it has been said in some of the other talks. If you have an executive big hammer to do it with that helps too but it's not sufficient because then just everybody's kinda doing it because the executive VP said to and nobody really wants to do it. You have to get people to want to do it. And that's probably the hardest thing overall. You also need some reason for doing it. When we started out getting a VM to deploy anything was a pile of tickets. It took week or two weeks. It had to go through review. Getting capacity was something that took a really long time. And agility was one of the first real drivers for using the cloud. It was a way to get VMs through a self-service API. And at first there wasn't really a whole lot more to it than we need capacity faster than we get it today or we're just not gonna do what we need to do in terms of deploying our projects. So really started out with sort of fast provisioning, not we want cloud, we just want VMs right now, right? But in order to make any of that stuff sort of work, it has to be self-service. You have to have automation. You have to have people buy into that sort of idea. And I'll talk about that a little bit later. Any questions about people right now? No. So I haven't said anything that anybody was not already thinking like, I think people know that people are hard, right? Yeah, so the question is, how do you change hearts and minds? No, I'm just rephrasing because you didn't go to the mic. So that is like standard people skills, right? So first of all, you have to have a relationship with other people in the organization. So you have to get buy-in from architects and engineers and the security team, although we've dragged them kicking and screaming a little bit. They've sort of started to come along and think more the way we want them to think. You get the architects thinking differently. And part of it is just simple repetition. You say, look, we wanna do this cloud thing. We're gonna start running the cloud. Hey, you're gonna have to cloudify your things. Hey, your VMs are gonna have to be ephemeral. Hey, you're gonna have to rewrite it so you're sharing nothing. And meanwhile, you're building a cloud off to the side. And if you have any support of any sort, you're able to sort of start giving capacity to people who start talking to other people and go, hey, you know, my app was up in 15 minutes because it didn't take a pile of tickets. So part of it is just doing it and convincing people that your product is better than whatever you were doing before. And it comes around. And part of it is also just allowing it to come back and for people to say to you the things that you've been saying all along. And you gotta sort of accept that the ideas have to belong to somebody else. Question is, how'd you get by the security people? So there's going past the security people and then there's not going past the security people, right? So it's not that everything was just totally ignored and it was the Wild West. It was more a matter of we were able to deploy applications that were in the core of the network. So we didn't have to meet the same, you know, we didn't have to meet PCI. So we said, okay, we're not even touching PCI. And today, two and a half years later, we're just barely going, how do we do PCI in here? So we skip problems like that. And because the sort of the nature of the multi-tiering of getting into that part of the network, you don't have the same requirements to have, you know, pan and a firewall and another pan and a layer of load balancers and another pan, right? You're able to just sort of deploy stuff in the middle and the standard security stuff that gets you to the edge into your cloud is already all there for you. So now we're having conversation with a security team which is like, more like, hey, we wanna put PCI in here. You know, and they wanna put a physical firewall all over the place. And I'll talk a little bit more. That's a technology problem, right? That's a little bit of a hearts and mind problem but it's more technology. Anything else? Okay, so let's move on to process then. So process is, you know, the pilot tickets that I was talking about. It's, you know, how do you give people capacity? How do people get access to the cloud that you've just built? You know, if people can't use it or if the process prevents it from working properly or, you know, there's too much red tape to all of it, that's gonna kill it as well, right? So we've got, you know, a product, I don't even, there he is. So we've got our capacity manager here and I won't point him out too hard because he hates that. So he manages our capacity, he plans the quota for people and he takes care of all of those sorts of things so that we can give people the capacity that we need. We're still sort of dealing with that sort of stuff because in a public cloud you have a natural sort of elasticity and downward pressure on what you're using, right? You pay for what you use and so if you use a lot, the public cloud provider goes awesome. We've got all this revenue to buy more hardware. In a private cloud, you give the capacity to people and they use it up and you go, yeah, that was the budget for the year and we're in Q1 and you're out of luck. So we've been really lucky in some of that stuff because that has happened and then we go back to, you know, the upper levels of Walmart and say, hey, you know, we need a little bit more capacity and so we build some more capacity and we've dealt with the downward pressure issue by getting through holiday and saying, all right, well, we made holiday traffic. You didn't need all that capacity, give us some back and so it's much more, a little bit more of a hammer than sort of a financial thing. So we're working on models to give us the ability to say, well, this thing costs us this much to host and to run and then we can do a wall of shame to say, hey, your service costs us, yeah, I'm just gonna say a million dollars, right? It costs us a million dollars and your next best competitor is like $500,000 and then like the bottom of the line where we're getting the most value is like $10,000 to run. These are all fake numbers, of course. And so getting the value against, you know, how much it takes to host it is the next sort of step of things to do to figure out, okay, well, we're getting our value out of the service or we're getting our value out of the software, you know, cause you can write really crummy software, right, that doesn't perform well. You can say, we're not getting business value from that thing and that's sort of the next step of things that we have to do. So we've got a cloud and it runs, but managing how people use it is the next phase of getting, you know, driving efficiencies and doing the thing that Walmart does, which is driving efficiencies. And so I already talked about this mostly, right? You've got to change the process. People coming to the cloud initially, like I said, just wanted speedy development or deployment, right? They wanted VMs now, they want 10 VMs now, they want 100 VMs right now. And even though there are people in the company who not that long ago were waiting on that pile of tickets and waiting a week for a couple of VMs and they can get it in 15 minutes, now we still get complaints, it's not fast enough anymore for them. So we're gonna have to figure that out. But it's all, you know, you have to change the pieces that prevent you from doing the interesting cool things because you have a cloud. One of the problems we have is that even though we do have the ability to launch VMs quickly and we have a pass layer to do all of this in, people are so used to grabbing capacity and this is my stuff that they don't readily give it up. And so we have stuff unnecessarily running and when you look through, you see some teams have, you know, four or five staging environments running. They're not doing anything with four or five of them and that part of changing the culture and changing the thinking. So the question is whether there are charge backs or whether we have plans for charge backs. We're not really set up internally to do charge backs directly. There are budgets, of course, and that's one of the processes that we probably have to change. So that we have a model where we can charge back to another unit or whoever's consuming it. Right now we're really looking at a showback model where we can say, hey, the hardware cost is X, you're using this amount of RAM hours and disk hours and CPU hours and whatever else and therefore you've consumed X amount for the company. So the showback I think will be step one and I don't know, can you change finance in that amount of time so that you can change your internal finance models? I think that that's kind of what we'd have to do to actually do real charge back. Yeah, showback, right, exactly. So another thing that we deal with quite a bit and there's somebody's head in the way right there. Thank you. So early, early on we said, hey, we wanna build an ephemeral VM cloud. We were rewriting all the applications. We had governance models that said, all right, we're doing this in Java. We want shared nothing, stored nothing, ephemeral kinds of workloads. So people were rewriting the applications that pretty much did this. So we've got this sort of concept that we want this sort of pets disposable VM model. You move along that line to things that last a little bit longer and you have the, gosh, I messed that up again. For the second time this week I did it backwards. So the cattle model are the ones you get rid of. I don't own any pets just so that you know, like so I've said this twice about pets. The pets are the ones, so we've got some pets a little bit that last a little bit longer. These guys are talking about the idea that if something's hard do it more often, that's an agile thing, right? And saying puppies don't grow up to be dog, puppies do not grow up to become dogs, they become dinosaurs, right? So if something sits around for too long, it suffers from configuration drift. People log into it and do stuff to it. It doesn't get updated and pretty soon you're afraid to touch that thing. And the problem with that of course is that at some point that thing is gonna crash and can you bring it back? And so we have caught developers, even though we have all of the automation, we have a pass layer and everything else, right? And they don't have to do that. We've caught developers who have logged into things and changed like names of the hosts or done other manual things to get their software running. And then something happens and we reboot it and it doesn't come back, right? So the idea here and these two guys actually did some actual research and collected some actual data. They had a talk originally called Cattlepats Mayflies, I think is what it was called. And it didn't make it in. I'd love to see the data and the talk and the paper. So if they hear this, get ahold of me, I'd love to see it. But the idea is in Mayflies is that you get rid of VMs after a set period. And we had started out with the idea that maybe it would be three months. We've had VPs say, look, replace 5% of your VMs a day. That means that your VM only has a life of 20 days. Now we're not there yet, but, because we do have some pets like thing in our environment, mostly that's a matter of getting that application to the point where it can be disposed of more quickly. And it does happen to store some data so they get a little bit of a pass on that. Given that everything's ephemeral and we don't have storage connected up now. Well, that's the sort of thing that happens if you let people get away with not doing that in terms of doing the automation and replacing things and making sure you're not making manual changes. You get these sort of hard little points in your cloud. It makes your cloud harder to manage. It makes the application uptime harder to manage. You know, and Netflix is doing things with like Chaos Monkey. So we've got a team working on something similar. Chaos Monkey is very, very tightly tied to a certain public provider in Seattle and we can't use it at all. So hopefully we'll get a chance to upstream that stuff as well. Not upstream, but release it. Talked about a little bit about providing capacity and you know, the problem we had is that there was so much pent up demand from the old way of doing things. You know, in the week it takes to get a VM and everything else is that we couldn't provide capacity fast enough. And so that's something to, it always seems that you do this with capacity. Whether it's bandwidth capacity or disk capacity or the 640K, that's enough for everybody. Which by the way, apparently he didn't actually ever say. But you can't sort of plan for enough capacity. At least that's the way we found, is that changing the process and changing the ability to consume these resources means that they get consumed even faster than they ever have before. So that's given our capacity team quite a time to figure out how to even plan for this stuff. So it has ramifications on your data center layouts. You know, your 30 year data center plan now only lasts 16 months because you fill it up with gear and you run out of cooling and power and the co-locages don't open anymore because you've got a rack cramped. This hasn't really actually happened to us. But it could, it could. So that's something that you need to take into account is that changing your enterprise or changing the way your teams do things really could radically change the way you consume resources. And it would be sort of great if all of your workloads could, you know, like just go to public cloud but you can't move data, right? So sometimes you just can't move your workloads. You can't move data. You just can't move a lot of data fast is the point. So in terms of deploying a cloud in the enterprise you're going to go into a data center that you have. You're going to have a network there. You're gonna have network gear there. You're gonna have a way that the network is laid out. You're going to have certain system standards that you're gonna have to work with. The security team is gonna wanna have things a certain way. You have all these teams who want to do things the way they've always done it. And so taking that into account our first deployment to the cloud really was taking all of these pieces from all the other teams bringing it together and deploying a cloud out of it. And that's one of the reasons that we chose OpenStack is that OpenStack is flexible enough to allow us to make odd choices about how we're going to do things. Would you necessarily do it this way? Version one, if you had completely, totally open green field and you were building a new data center, maybe not. But the idea behind OpenStack and one of the values that I think that it brings to an enterprise is that you can take the bits and pieces you've got and do something usable with it. Call it cloud version 0.99 or version one or whatever you want. It's your alpha cloud, but it's got people using a cloud, doing real workloads with it, bringing real value to the organization. And you didn't have to change everything you've got in order to do it. So we used existing IP schemes. We used existing everything. You fit sort of at that time when we started two and a half years ago, there was no networking service separate from Nova Network. So we just used Nova Network. We just slid a flat network in along with all the other VLANs. And the networking team didn't get too worried about it because it looked just like all the other VLANs. So we were able to sort of just slide into an existing data center. No, there were no virtual routers. We're still not using virtual routers today. So we're still in terms of networking, we're still sort of at that, use what was there. It's a matter of now changing hearts and minds again and sort of starting over with now, we've got to change the networking. Of course, we have a security slide, right? Because you can't forget security. Being a retailer, we accept credit cards. So we have to do PCI. It turns out that also we have a pharmacy. So we have to do HIPAA. And it turns out that we're Walmart. So there's something that we have to do because there's a regulation for Walmart. And then of course we like to put our own twist on things. So PCI starts out with something that is based on the fact that everything is physical. And so you're talking to your security guys and saying, okay, well that was awesome and everything was a bare metal node. Now you got a VM. We can't do that physical thing to that virtual thing anymore. So we need to change how we're going to do PCI. We're actually going through that right now. So I think we've convinced them that you can't stick a physical piece of hardware between two VMs and that you can't watch every single packet that you got to choose the stuff you want to look at that yeah, security groups aren't perfect. IP tables aren't perfect, but they are stateful. And they can do the job of keeping things from getting to other VMs. So it's an ongoing conversation with security. So probably it may have been easier to engage the security team a little bit earlier and say, hey, this is what we're doing. But at this point now they can't say you can't do OpenStack because that ship has sailed. We're doing OpenStack. And now they've got to kind of figure out, well, how do we make the company safe for the different requirements that we have? So you have to meet PCI. You have to meet HIPAA. You have to meet, you know, EU security requirements on data and that sort of thing. So you can't get around them so you have to sort of embrace it. And hopefully you can get your security team to embrace it with you. So we've had some great conversations with some different pieces of our security team talking about, well, how can we actually make this happen? Anything about policy? Yes. Have we passed PCI with OpenStack yet? The answer is we haven't tried yet. So that's next. We're really sort of still defining some of those pieces and figuring out what capabilities we need to bring into our cloud in order to meet those requirements. So the answer is we haven't tried yet. We're working on it. Okay, so the question is, so we've invested in this technology, I assume you mean OpenStack. And the question is how much? And that's one of the areas that I can't cover. So we, I don't think it's a secret anymore. We are working with Rackspace. Okay, so now I'll preface this. This is my thoughts. These are not Walmart's thoughts. I really like Rackspace for what we're doing in globally commerce because we want to be able to contribute upstream. So Rackspace has an open distro and an open deployment method. And you really just sort of buy support to get where you need to be with the distribution, right? If you surpass them or some reason you argue with them about how they're doing support, I mean their support is awesome, don't get me wrong. But if you decide that you want to part ways with them, there's nothing proprietary about the deployment or about their distro that locks you in. So as far as WEC is concerned, as the e-commerce is concerned, that's the direction we've taken. Now, Walmart's pretty big, right? And we have different operating needs and different parts of Walmart. So if we also have OpenStack that happens to have VMware underneath, that would be because we need VMware capabilities underneath it. We have deployments that are now owned by EMC and so we've done work with other companies. It's still the OpenStack API over the top and different things underneath to give us different sort of accelerators or capabilities in different areas of the company. So I've said this before, if anyone's surprised that Walmart's talking to like the top five of anybody in any area, you have no idea how big Walmart is. And it's important to us to use the right technology for the right problem that we're solving. But so that said, we're looking at all sorts of stuff. We're members of the Helion early adopters. So we're giving feedback to those guys as well and working with them as to how much we've spent. Actually, you know what? I can answer honestly. I don't know. But if I did know, I wouldn't give you the number. So anything else? I heard, yeah, first, yeah. Not between different vendors, but because of the way we've got things laid out and this will come a little bit more in the technology. But I guess it's a process question that we do have things set up in such a way that we can take down part of our production capacity and do things to it. So are we done yet with the Juno upgrade? Almost, so we're almost done. We're more than 75% upgraded of our production cloud capacity from Havana to Juno. We skipped Ice House. Almost all the team is here. So that's how well it's going. Walmart.com is up. So we're doing that, I hope. I'm sure it is. We've had a lot of success rolling upgrades, destructive upgrades, which means that we've destroyed all the VMs running on it and not had service impact. And you had a question. So the question is how do we share data across areas and do sort of the stateless thing that I was talking about before? And we do it a number of different ways. So we do have databases that are physical databases from a large database vendor that you all can probably guess, right? So those are not part of the cloud and they're never gonna be part of the cloud and those are kind of the dinosaurs in our environment, right? We are talking with Plumgrid about some interesting things that you might be able to do to pull the physical nodes into a neutron network but that's all theoretical and physical, theoretical and some other word that means theory as well right now. We do use Swift. So if you go to Walmart.com and look at a product, that image comes in through a CDN. It goes through our optimization and intermediate caching layer and the origin for all that stuff is a Swift deployment. And I think those are two primary sort of storages of data. Well, those are two primary storages of data that have anything to do with the cloud. We have a gigantic Hadoop cluster and there's a lot of data in there and then we work with other stuff. We've got a lot of mainframes too, so. Sure. So anything we would do differently or decisions that we regret. I don't think we have any real regrets. Like it's been surprisingly successful and smooth. There have been some battles with the application team. Sometimes they wanna do something that's not very cloudy or not very operationalized. So we do have those battles still but I think it's a healthy conversation, right? Because it teases out kind of what we wanna do this thing with this new technology. Okay, how ephemeral is it? Well, it's not very ephemeral. Okay, so we run some of that stuff. What would we do different? I mean, the world has changed a lot. So I started with an Alamo deployment which was based, that was based on Essex, right? Anybody? I think it was based on Essex and we still have three nodes running Essex. We're not taking any workload. So we went Essex, Folsom, Grizzly, Havana, Juno in two and a half years. So if I could do anything differently, I would have started with Juno two and a half years ago. But I don't think we made any major missteps that were not steps that you sort of had to take along the way as we made it, right? So you had to start with Neutron Networks back when we did that. As soon as Quantum was available, we started using Quantum and then we did the terrible database thing and we moved to Neutron, right? But we knew that there were sharp edges in parts of Neutron, so we didn't bet the farm on, hey, we're gonna have a million tenant networks and we're gonna count on V-Routers to do it and we're gonna have floating IPs into everything. Cause it wasn't HA then. We had the flat networking model that was working great for us at the time. And so, it'd be great to sit back and say, yeah, I would have loved to start two and a half years ago with everybody going, I get cloud, I know how to write an app for it, I can do a three tier, multi-tenant, private network, total virtualization play from day one, but that's not the world, right? So I think we've done a pretty good job of going from we have no cloud at this company to we took holiday traffic last year. 100% uptime, by the way, and not all of our competitors can say that, so. Over here, sure. So the question is, oh, I've got five minutes to get through my next section. The question was, what about monitoring and that sort of thing? So we've got sort of, did you say salometer? Troubleshooting, ah, yeah, okay. So, this is something else that we talked about and I think the operators have to come together and say, hey, look, OpenStack is an operating system. It's not necessarily a monitoring system or a troubleshooting system. We have fairly standard sort of enterprise-like monitoring. We've got Nagio stuff and all the standard stuff you'd expect to see in a knock, so you can do all the standard monitoring with that stuff. In terms of troubleshooting, you get trouble reports and you try to make your alarms as useful as possible, but we're still in a world where people kind of log in and take a look and see what's going on and what the process is or we just ripped out OVS, but the last talk was about how OVS kind of, you kind of have to learn all the ins and outs of running OVS in order to run OVS. Let me see how fast I can get through these technology slides. So, technology. Everybody's going to talk about iterating and we're doing the same thing as well. So, we deployed a cloud, are we totally happy with it? No, we weren't. So, we changed one thing or another. We upgraded the software. We pulled performance patches back releases. You know, our goal has always been to sort of pull back and leapfrog over so we weren't doing things like maintaining forks of any sort. We're changing what we're doing with hardware. So, you know, you take whatever the model is that happens to be in the data center and you use that until you can do the next batch of buys and then you can iterate on your hardware design. We're going to be iterating on our network design and changing the way the network works so it's more of a cloud ready network. I talked a bunch about the non-cloudy apps. Non-cloudy apps cause constraint problems in your network or in your cloud. You know, there are things that you can't just kill and they'll take care of themselves. You've got to take care of them a little bit. Take care of them a bit more like a pet. But we've been trying to treat these things a little bit like a pet you don't like very much. So, we kick the puppy a little bit and swing the cap by the tail and all that horrible stuff. So, it's totally possible to do and we're forklifting some other applications in the cloud that are not cloud applications. And you really just sort of have to make sure that people aren't doing horrible things with it like having, you know, hard-coded IPs or system names or things like that. So, contain failure. I think this sort of speaks for itself and everybody has talked about this is that everybody's basically deploying multiple clouds or regions or whatever else in their data centers. And we're doing the same thing. So, we have completely separate deployments so that we can completely lose an entire pod region and we don't lose control planes for any of the other ones. And that's basically to deal with the sort of problems that that same large public provider has when there are outages on one coast, your VMs are all there, but you can't do anything to them. And losing an entire data center worth of capacity is more catastrophic when you're running your own private cloud. So, right now we're taking care of that but with just completely separate clouds. I love this slide. Nobody seems to know who did it. It's anonymous. But the idea is you don't have to deploy a complete public cloud day one. Deploy what your people need at the beginning. So, we have a very large compute cloud and there's no storage attached to it. There's no heat attached. Well, there is heat now but we didn't start out with heat and we got away with no storage as well because we didn't have really used cases for it. We didn't really need it. And the developers came to us and were saying, hey, we love this storage thing. We love these block volumes and we can manually create them and configure them and clone them and attach them to VMs. And that was the end of that. We're not gonna let you do that because they were gonna cheat automation. And so, we have all these different clouds. We have all these processes. You can't drag all your developers and make them cloud architects and cloud application architects day one. So, we've sort of glued everything together with our PAS layer, which is OneOps. OneOps.com has a lot of information about it. It's sort of public. We bought them about two years ago. So, the fact that we've got this capability is not really a secret. And we'll see where things go with those guys because we're always adding stuff to it. And I don't have a lot of time for the future, but, and I'm gonna skip the questions for that section because I'm basically out of time, right? We're gonna try new things and I'm gonna go through these super fast. We're gonna try new things. So, V3 authentication, load balancer stuff. We've been working with load balancer companies. Iteration, always gonna do iteration over and over again. SDN, so we're gonna play with SDN. You can take a guess about some vendor that maybe we've been talking to about some technology that they have. We're really interested in some of the ways that we could use that technology to stitch together tenant networks across the various clouds that we have. Because right now you'd have multiple tenant networks in each of the different clouds. How do you do things between those tenant networks? And there are some capabilities that you get where you can stitch together those tenants. Storage services, we're gonna do that. We're gonna do stuff with LBAS. I asked a bunch of questions, or you guys asked questions and I answered them. And that's the end, thank you very much. I'm only a minute and a half over, so.