 Welcome everybody here. My name is Patrick Lee. I'm a software engineer running Cassandra here at Walmart. I've been working on Cassandra for the past eight years. I'm Andrew Weaver. So welcome. We're part of the Cassandra team here at Walmart. So Cassandra goes back a long way at Walmart. We started back in around 2011 in the 0.7 days. So today we're going to talk about multi-clown at Walmart. So this is just going to be kind of what we go through today. So I'm going to show you a little bit about kind of the current state where we are with Cassandra at Walmart, why we chose a multi-clown strategy, a little bit of detail on what that multi-clown strategy is. And then we'll try to get into some Cassandra specific stuff because I assume those that are joining this talk are more interested in the Cassandra side of things. And then some challenges we face and we want to leave some time at the end for Q&A. So just a little bit about our current footprint of Cassandra at Walmart. So we've got over 300 production clusters today, multi-pedabytes. Our cloud journey really started in about 2015 where we started migrating a lot of our bare metal clusters to the cloud. Since then we've had like 20 times growth of our Cassandra offering at Walmart. And every year it just continues to grow. Our entire footprint is 100 percent DMs. We've got almost every one of our clusters upgraded to Cassandra 4 from 311. We've got a couple clusters running 4-1 and we've actually got at least one that's running the 5-beta so we can get our early start on testing. Okay, so why multi-cloud? Multi-cloud is a little bit of a hot topic, maybe not quite as hot as AI, but so multi-cloud at Walmart, why do we want to have a multi-cloud strategy? So what that means for us is being able to use multiple cloud providers to run our workloads. So multiple public cloud providers as well as our own OpenStack private cloud. So we're a very, very large company and so that means we have a lot of different use cases. So yes, a lot of retail use cases, but there's also just a multitude of other use cases. Finance, HR, inventory management, all the things around retail and then compliance and just many, many different types of use cases in a company of our size. And so those use cases are going to have different needs. And so some of the reasons for going to a multi-cloud strategy, we do want to prevent vendor lock-ins. So obviously if you're tied to a single cloud provider, you know, you don't have a lot of options for moving or moving to another cloud provider that may suit your needs better, could be difficult. And along with that, we want to enable best of breed. And what we say, what we mean by best of breed is looking at the different public cloud providers and the feature sets that they offer, whether it's part of their IaaS offerings of compute and storage or looking at the platform as a service offerings, such as databases or other services. Like I said, we have a lot of different use cases. And so some of those products from one provider might work better than another provider for a specific use case. And so yes, while you're trying to prevent vendor lock-in, you do want to enable the use of some of these products. Yes, that's kind of an intention a little bit. But with the broad set of use cases that we have, being able to select the right tool for the job is very important. And then obviously, driving down costs. So if we can force our cloud providers to be competitive with each other and compete for our cloud spend, that helps us reduce costs. And so our multi-cloud, so the process that we kind of went through, first off, you just have to modernize your applications. Really the stateless applications, business applications, you have to get those ready to be able to be deployed in a portable fashion. And so part of that is Kubernetes and containerizing applications. And at Walmart, that's what we call the Walmart cloud-native platform. That's Kubernetes with a lot of automation. So developers are free to just write their code, just add a few lines of YAML and be able to deploy that application anywhere. I'll let you take the rest of this one. So that's the breed. So kind of what Andrew was talking about before. Obviously, different cloud providers are going to have different strengths. Our private cloud is going to be quite a bit cheaper. In public clouds, you've got your compute, you've got your storage, you've got the other pass offerings. So being able to pick and choose and being able to, for us, for having Cassandra to be able to deploy across any of them to meet the application needs and to be local to the application has been a real key factor. As far as pass products go, yes, some pass products are good. What we've learned is it's very important to read the fine print, do a lot of testing, and really get to understand those pass products. A lot of the limitations that you might run into for some of those might not be obvious at first. It might sound very great, but once you kind of get into the thick of it, that's when you start uncovering some of those. So it's best to spend that time up front and try to get a very good understanding of what those products are. Determining your data strategy placement. There could be different security considerations to be able to choose a particular cloud provider for certain data. So good idea to have a good placement strategy to figure out what data would we want to place in what provider cost could be a consideration for that as well. One of the very key things for us for Cassandra to be able to deploy across all these different cloud providers is having a central consistent observability stack. So metrics. If I deploy Cassandra, I need to be able to see the metrics. If I have a cluster that spans multiple providers, I want to be able to see a holistic view of that cluster to know how that cluster is behaving across those different providers. So our multi-cloud strategy started 2019. We've been running through 2020 up till now, and it's been very, very successful for us. We've been able to reduce cost and really accelerate the developer experience. So what is this thing? We keep talking about this multi-cloud. So for us in multi-cloud, we're using our very large private cloud, Azure and GCP. So what do we do is we get these different cloud providers together spread across the United States, and then we get those very close together so that there's milliseconds in between each. What this allows us to do is the application can be deployed in Azure, for example. Let's say they're using some of the Azure technologies, and that's the best fit for their application. So we're able to deploy the Cassandra clusters, have rings of Cassandra cluster in Azure, but then maybe they have some other use cases. Maybe they have some more data warehouse, ML type of workloads that we're going to run in our private cloud or in GCP. Maybe those are going to be more storage density, so we're going to deploy it in our private cloud to save cost. So I can have a Cassandra cluster that spans both our private cloud and our public cloud regions. So just have rings of Cassandra kind of all over the place to service different workloads. This allows us to be able to dynamically change and move in between those. It gives us the flexibility to take advantage of some features, let's say in private or public cloud, where I can more easily switch between let's say an 8-core machine to a 32-core machine and able to vertically scale to add more CPU for a particular event. Let's say they're going to have Black Friday event and we're going to have a surge of traffic. So instead of vertically scaling out a cluster, I can just horizontally scale out a cluster. I can vertically scale a lot faster. So given those choices in the regional triplet, we have multiple cloud providers. So how do I choose where I want to place my data? These four different factors, these are the main factors that go into making these choices. So obviously, you've got public versus private. And so in the public cloud, of course, cost is going to be much higher than if we run our own cloud. We can run our own hardware a lot cheaper than using somebody else's. The feature set, like we've talked about, you got everything in the kitchen sink in public cloud. The private cloud, it's a much smaller feature set. And of course, the scalability as well. In public cloud, if you have a bursty workload or you frequently are scaling up, you're more able to do that in the public cloud, whereas private cloud is more of a steady state type of deployment. And then performance. Performance, in order to keep costs low, our tech refreshes in a private cloud probably aren't as frequent. And again, it's not. You don't have the broad set of options that you typically have in a public cloud. So those are some of the factors that go into how I might choose to deploy a cluster, or maybe if I need to move a ring in a Cassandra cluster from one provider to another, we have that flexibility. And yeah. And so like, for instance, on performance, Azure has 11 different virtual machine series. They have five different managed disk offerings. Google has 12 different machine series, and you can customize them. Whereas in private cloud, it's a lot more, it's a much smaller set of choices. And so some Cassandra-specific things, considerations, if you're looking at a multi-cloud strategy. First, I'll just let you know, it was pretty easy. And this is a bit of a callback slide to our presentation in 2015 when we were talking about bringing Cassandra into Walmart, into a large enterprise. And we had a slide that just said, it was hard. It was like super hard to change people's minds and to build a case and get that in. But here we are. Eight years later, and we're able to say it was easy because for Cassandra, multi-cloud, multi-region, it just works. So as long as you've got that network connectivity between your cloud providers, deploying a cluster from one provider to another is just adding another ring to that cluster. So the native multi-DC replication is just business as usual. And obviously, you have to invest in automation to do this. So for us, we have a cloud deployment platform called OneOps that abstracts away all those differences between cloud providers. So we're able to leverage that. So you definitely have to have some automation in place. And benchmarking. Benchmarking is very important. We've done many, many different rounds of benchmarking, different configurations, and different providers, again, to just achieve that parity between providers. So because we want to offer a similar user experience at a similar cost, so particularly in the public cloud, you don't want one provider to be significantly more expensive if we can control that by being smart about what VM flavors you choose, the storage configuration, things like that. And so Cassandra, it just offers a better user experience in this type of situation than other data platforms. Like I said, it's business as usual. It's just another ring. It just happens to be in one cloud provider or another. And of course, we have to give our network engineers a lot of credit. So we have some really good cloud network engineers at Walmart that make this all work and make it look seamless from a network perspective. And then from a developer perspective, it does look a little bit more complex. I've got to try to co-locate my app with my data, and there's a lot more choices out there. So understanding what your environment is and where you're deploying and that you configure your driver to talk to the right endpoints and use the right local DC name and things like that. You just have to make sure you get that correct. And then some of the challenges that we faced. Like we said, these hardware differences between cloud providers do matter, VM flavors. So you really have to look at what you're getting in a particular VM flavor. Are you getting full cores? Are you getting hyper threads? Do you need to use a 32 core machine in one provider where in another you might use a 16 core? Discs also very different strategies if you look at Azure and Google and slightly different cost models and how those work. And so you really need to know where the throttles are. So most cloud services, including VMs and storage, they tend to have limits in different ways. And GCP and Azure do their storage. Those throttles are implemented very differently. So you have to understand that. And really it comes down to benchmarking it. So run those tests and try different configurations to understand what works and what doesn't and what gets you the most equivalent performance at a similar price between all of them. Because you really don't want any surprises when your customer goes and deploys to one cloud provider or another and then their cost shoots way up. So again, a similar user experience is very important. I'd say keeping up with the changes in those cloud providers too. So we have had to spend quite a bit of time up front obviously trying to evaluate the different VM SKUs, the different disk and everything to try to match that parity from where we started with private cloud. So we went to private cloud, we started moving into Azure. How do I get a similar experience, similar if not better experience as I move? And then again, when we started out in GCP. Okay, well then several years past, now there's what we were DSV2s at one point in Azure. And now they've got V5 hardware and everything's kind of changed. So over the past, let's say over this past year, we've been going through and reevaluating a lot of that stuff and then migrating to newer hardware. So again, we had to kind of go back to the drawing board, spent a lot of time up front trying to reevaluate everything that they have to offer. What's the right configurations? What's the right disk types? Is things the same as what we had or do we just need to switch to a different VM? Then when GCP comes in the mix, their offerings are very different as far as how the caching works and how the IO works. So how are we able to get the same for the same core account for the same storage? Or do we have to kind of get creative and try to add in some caching, some disk caching like some LVM cache is kind of one of the things that we kind of looked into to try to help better offset and get the performance on par with Azure. Okay. All right. Well, that is all we have as far as presentation. We can move into Q&A. If anybody has any questions, we do have a microphone up here so that people would be able to hear you on the stream. So we'll open it up for questions. Sorry to make you have to come all the way up here, but I don't know. Come on. You're first. Thank you. So my question is you mentioned that you have multiple cloud providers for the Cassandra ring. So for the same data center, do you use only one cloud provider or within a data center? Let's say you have three replicas. So you have one copy in Walmart. Another one is Azure. If that is the case, then how do you deal with even a smaller latency difference between the zones between different providers? Okay. So when you say data center, you mean Cassandra data center like the Cassandra. Okay. So there's different ways we can do this. So if we have a Cassandra data center, so one ring of a cluster, that is going to reside within one provider. So we're not doing anything. So when you see the different cloud provider logos in these regional triplets, so that would be one ring within one provider. So we're not doing anything like, we're not treating them as racks or anything like that. So that's been the best way to do it. I don't think you really want to spread a ring across multiple providers, but use multiple providers by having different rings for each provider. And the latency there, it's single digit millisecond latency is the goal between any provider within a triplet region. What do you use for observability? You mentioned that that's a key constraint you have to work. Observability. So we have an internal observability team. And a lot of that is Prometheus based and Grafana. So we're using the M3. The M3 is the data store, the exporter, the current exporter that we're using for Cassandra is the Instacluster metrics exporter. That seems to work the best for us. Yes. Thank you for interesting presentation. Simple question. How do you design the benchmark? Do you use a typical mix of YCSB or something like that? NoSQL Bench has been used a lot. Back in the early days, it was Cassandra Stress, which I still love, Cassandra Stress. It's a great tool. But NoSQL Bench offers a lot of flexibility. I think there's a talk today or tomorrow. Jonathan Schuch is giving that talk about NoSQL Bench. It's a very, very powerful tool. I would recommend setting aside some time to really look at it and understand it before just trying to run with it. But yeah, NoSQL Bench works really well. Okay. But I guess there is some gap between the real workload and SQL Benchmark. So how do you overcome? A gap between we are workload and SQL Benchmark. Yeah. So what we've got is what we start out with is some basic benchmarks, mostly around key value lookups. I mean, we try to benchmark different payload sizes, right? So that is not necessarily a real workload, but that gives us a baseline that we can use consistently across those cloud providers so that if I know, if I deploy this type of workload in private cloud, here's kind of the performance that I get. And then we use those same benchmarks every time we do a different cloud provider or do a different VM skew. So that way we have at least a common baseline. And then with that, so like when we were doing our Cassandra for migration and benchmarks, that was also evaluating some additional VM types. We ended up partnering with one of our application teams where we took one of their real use cases and deployed it alongside Cassandra 3, Cassandra 4 in Azure and private. And then we were able to have them run their applications while we were taking care of the backend database, they were running their application and we partnered together to put together kind of like a white, an internal white paper and stuff to figure out performance cost. And I did actually kind of match back. We saw similar gains, similar trends with our static benchmarks that we do. Okay, thank you. You guys mentioned about automation built to add new token rings, right? Do you have any automation built to scale token rings? Yes. So in the, like I mentioned, the tool that we use is called one ops. So that's a home grown tool. It's open source. But that is an automation platform. And part of it is it has automation for Cassandra. And so adding a ring is captured in that and then scaling up scaling out horizontally and scaling up vertically is all within that. We have a few little add-ons that we use in there to drive that automation. But we do use VNodes. So that ownership is a little bit easier to deal with. But we end up doing, we've gone as low as four, four VNodes. Thank you guys for the presentation. So the question I have, you guys talked a lot about the infrastructural side of things. So what about like application development side? So you said that Cassandra was easy. And I believe before Cassandra used a lot of like traditional relational databases, right? How was that like transition from relational model to non-relational model? Was it like easy? Like I'm trying to put every single app and you know, model to Cassandra now? Or are you still like debating between depends on the use case? So as far as data platform within Walmart, we have relational offerings and no SQL offerings. And so there's really just, if your problem is, can be solved with a relational database and maybe you have some needs for transactions. And you need to be acid compliant, which of course this is changing with 5.1. But going with the relational database is still, I mean, in my opinion, still a valid choice. We don't necessarily want to Cassandra all the things, because part of what we do is when we onboard an application team is we do ask a lot of questions upfront. And one of the major questions is going to be, can you tell me all of your select queries that you're going to run, right? Because Cassandra applications live and die by their data model. And if we can't help them design a proper data model, then it's not going to be good for either of us, right? Because things just are going to fall over. You're going to have issues that may not even be apparent upfront. But over time, you know, you're going to have a problem. And if it's data model related, it's really hard to change quickly. So yeah, there's definitely space for other databases out there. But we do have a lot of Cassandra use cases, and we do try to work with, as part of that onboarding process, really design the data model, make sure they understand what the consistency is like, eventual consistency that they can tolerate that, and that they're configuring their apps well. And really, you know, basic rule of thumb is use the Cassandra, the data stacks, typically it's data stacks Java driver, version four and up, specify the local data center, and then take all the other defaults. And you will be successful 99% of the time. So that's, that's kind of what we work with our application teams on. How could that answer your question? Yeah, partially. So my another question. So how do you like what's your approach to maintain data consistency? Let's say you have like multiple tables. Are you trying to spread data like across multiple tables? So it's trying to keep data, let's say in a single table, create like secondary indexes or, you know, something like that. Okay, okay. So yeah, if you have, so you got a denormalized model or sorry, or whether or not it's used, if you use them, no, no, we do not use materialized views or secondary indexes. We actually try to tell people, you know, yeah, secondary indexes are a red flag. We do have some clusters out there that have secondary indexes and apparently it's working fine because we don't, you know, we don't get any alerts or anything. But yeah, materialized views are disabled everywhere. Bad experiences really probably not a fully baked feature at this point. I actually had it crash a cluster. Yeah, just because they decided to create it on top of a fully, a full table. And it just wreaked havoc for a while. So, so yeah, there's still those, those sticky points of if I've got to have update two tables at the same time, right? Do I do an atomic batch, right? And we really try to refactor those use cases out. Item potency is huge. If you get into this read, modify, write cycle, that may work for a while, but it doesn't scale as well. We get people trying to do a lot of, I just want to store a JSON blob, right? And that forces you into that read, modify, write cycle. And it forces you to read the entire payload every time. So we talked about benchmarking different sized payloads. That's very important because you'll run into different, different limits if I'm doing a one K read every time versus I'm trying to read 16 kilobytes of data, right? That's a very different use case. And it's not going to scale as high. So, yeah, there's a lot of data model and use case things that we really try to work out up front because you can run into trouble pretty quickly if you don't put thought into that up front versus third normal form and a relational database, you know, it's, it may not give you some millisecond latency, but it's comprehensible and it's probably going to work for you. But if you have, you know, terabyte plus scale needs, I mean, you probably want to go with Cassandra. Thank you. It should be it. Yeah, I think, I think we're at time. I don't know. Let's take one. I'll make the call. We can take one more question. Thank you. If you can, could you elaborate on the use cases that maybe you've seen Cassandra like shine and then maybe some where you thought it might have been a good idea and then it turned out to be not so, not so high. Let's see. It's easier to name the bad ones. So anything with really large partitions? Unless it's like time series and use time window. How large is a large partition? Okay. So we're working, I'll tell you right now, we're working through one. We've got the table is a couple hundred megs and like 19 megabyte max partition size. And we're getting queries of boarded due to tombstones. Yeah. So really tiny table, really tiny partition, but you can't read it because there's just so many tombstones. And it's just the way the app works. And we got bit because they did have, it was a zero GC grace, I think it was a zero. It was really low GC grace value and we got zombie data. So this is a really tough one to solve without just upping the tombstone limit. But yeah, I mean, like I mentioned, the JSON blob use cases and a lot of people complain about their latency and go to dig into it. It's like, well, let's do some optimization here. And those that have optimized and either modeled out as columns or really tried to shrink the size of that payload. They've seen big payoff in that reduction in latency. It's a lot more consistent latency as well. Treating it as a file system. There's ways that can work, particularly if you chunk up those files and, you know, spread them across partitions versus just dumping a large blob into the database. But, you know, I would argue object storage is going to be a lot easier to deal with and a lot cheaper. I don't know, other use cases, good ones, bad ones. I think that's been good. I'd say the worst, the worst large partition that I've seen is a terabyte. We've had one of those that I wasn't seeing, but they never read it. So yeah, I don't understand. Heavy usage of like check and set and bad consistency levels, you know, trying to do each quorum all the time with no retry or anything. Yeah, if you have any questions, hit us up. After the fact, we'll be glad to sit down and chat. Yeah. All right. Thanks, everybody.