 Welcome everyone. This is zero downtime migrations or in other words, migrating applications between Cassandra clusters of whatever variety with no downtime. We'll get into the nitty-gritty details of what that actually means in just a little bit. My name is Jeremy Hannah. I've been around in the Cassandra community for a while. I would have made it to the first Cassandra Summit in 2010, but my daughter was born. Worked with Cassandra just back that far, working at data stacks for about 11 years now, but in the Cassandra community for about 13 years, I guess. Hi everyone. My name is Wei Dan. I've been with Cassandra community and data stacks both for just a little over 10 years now. We're going to be discussing some technology that some tools. So there's what's called a ZDM proxy or zero downtime migration proxy and a set of data migration tools that a number of engineers have worked on. We wanted to just not take all the credit for all of this. We're just the people that are actually here physically present. But the list on the right that you see are a lot of engineers that have put a lot of work into the data migration tooling and been in the trenches doing a lot of these migrations over the last couple of years. And the list on the left are those who are primarily focused on what's called ZDM proxy, the tool itself to help with the no downtime part, keeping two clusters in sync and Alice Lottini and Joel Rice in particular. I've done a good share of that work starting about three years ago. Yeah, I see Stefano is also in the audience. And same with Brett somewhere out here too, in the back. So yeah, we have some if you have any additional questions about certain things. So we'd like to just start off with what does it mean to be a zero downtime migration, how you might define that. I want to do it from the perspective of a user or the application developers. Because this is the process of migrating applications from an origin to a target cluster, origin cluster to a target cluster without user awareness or service disruption. So if you're on a banking app, you're on a social site, you're on whatever, you're on like a movie streaming service or a game or something like that, you shouldn't see any blip or blip, you have to have some sort of like, by the way, at 3.30 this afternoon, something's happening or whatever. They should have nothing like that. And that's not to say that you might not have rolling restarts of your application servers or something like that, just as you would with something that is stateless or something that can refresh user connections and things like that. So something might be happening under the covers and we'll go over what those things are. But from a user's perspective, that's using your app or your website or whatever IoT thing. They should have no awareness of what's going on with the covers, except to say that their service is predictable and performant. Yeah, so one thing that I could call out here is that when we say origin and the target, which you will see a lot in the diagrams and needed to follow up in some of those subsequent slides. So origin, hopefully that's self explanatory. That's basically your migrating from the origin cluster. And then your target is the data you will be going to. So yeah, just wanted to make sure that you're clear on the terminology. So the scenarios and use cases, we originally designed this for migration to data stacks has a managed database called Astra. And it's a managed Cassandra and people have had Cassandra clusters that they wanted to migrate to that. And so we wanted to create tooling to make that simpler for people. But as kind of like added side benefit from that is that this can really be done for. And we've used it for a number of other reasons. So for instance, and this builds upon what you already have in your tool box, right? So if you have a Cassandra cluster, you're trying to say, I wanna be able to burst from my on-prem into the cloud because I wanna go into the region or I have this specific event. You can add a data center in a cloud and have it be the same cluster. And you just kind of burst out to that other area with other application servers. This adds something to that capability because under certain circumstances, you don't wanna do that. So like if I'm on-prem and I'm going into a cloud provider, there's kind of some detangling coupling things that you might have. So like authentication or other things having to do with like user management or infrastructure and networking and things like that. That if you don't necessarily wanna couple the two into a single cluster and having two separate clusters that are decoupled completely gives you a lot more freedom and we'll go into how that works. Or upgrading from an older version of Cassandra to a newer version of Cassandra. If you're on 2.1 today using CQL, you might wanna go to 5.0 tomorrow when it gets released in GA and you've got to prove it out. And so not having to go through, if you've been in that situation going through a lot of versions of Cassandra, upgrading in place is painful. If you have to go like three steps and then do the upgrade as the stable's along the way and things like that. This helps with that as well because you have two different clusters. You have a lower risk because you don't have to say, okay, well, if something goes sideways, what is my state and how do I unwind from that? It's more like, okay, we have the target and it's completely independent. It's starting from scratch and you have this origin cluster and if I ever need to roll back, I'm good. And then you don't even have to do the full cluster. So like you say, I wanna separate out this one app that's kind of like a nuisance app or a noisy app. It's really mission critical and we wanna keep it on its own hardware. Let's split that one app out. And the ZDM proxy will allow you to do that or consolidating either direction. When it comes to compatibility, we've tested it against everything you see here in one form or another. I think most of these in production, going from one to the other. It speaks the native protocol versions two, three, and four. And so anything beyond Cassandra 2.1.7, I think. Or similar data sacs enterprise version, associated data sacs enterprise version. And it goes to version four. Version five, protocol version five, CQL protocol version five is in beta until Cassandra five. The ZDM proxy is able to downgrade, kind of like negotiate the protocol down to version four if it needs to. So it's not like it's a critical thing that it doesn't have version five in there yet is just one of those things that we haven't gotten to yet. Sanders five's not out yet, but that's one thing that we wanna add. So pretty much anything that speaks CQL of any sort should be able to do this. And has this been done before? Yes, we've been doing it for about a little over two years. It started development over about three years ago. Doing this for a little over two years with a number of customers. We put it into categories of kind of verticals because they oftentimes will have different tolerances for security, for SLAs, for throughputs and all of that. And so just to give you an idea of, okay, well, has this been done for a lot of clusters and you have like lots and lots of clusters for individual companies or you have like, my favorite is like a gaming company that released a game, like an online massive game with crazy bursty traffic. And they were using the proxy to bifurcate traffic during the production launch. And I'm like, that's a really nice validation that this is not gonna fall over. And also the data warning is like 20 plus terabyte, right? Yeah, so in the background, you're migrating all the historical data. And we'll go through some of the, so we wanted to kind of give you some context as to why and then the tools that are involved. So the proxy that I've been talking about, it's a lightweight CQL protocol handler wrapper to be able to do this bifurcation of traffic and keep things in sync. It's one job is to say I have these two clusters and I need to keep those two clusters in sync after time X. So it handles all the real time incoming traffic and at each request it says, I'm going to synchronously make it so that this cluster and this cluster receive the exact same updates of the exact same consistency level and it's synchronous. And so, and it's common, it's written in Golang. It's deployed as a Docker container and it's all open source. And so if you have any issues or whatever, it's an open source project out there. And yeah, so why would we want to have something dedicated for this? And it's stateless. That's the other important thing. It's stateless so that you're not having to worry about. It does have to read to understand whether it's a read or a write to do the appropriate thing. So it does have to kind of, so in terms of end-to-end security, you can have client to proxy encryption, proxy to clusters encryption, but it does need to be able to see what the requests are actually trying to in terms of reads or writes. And so why would you want to do this? We've seen at other times when people go from one completely, one data store to a completely different data store where they might say, okay, well, let's do this, something Kafka, something with like queuing and things like that, or you want to bike for Kate within your application. And it puts a lot, I mean, it seems like a simple idea in theory, but it puts a lot of burden on the application developers. And if you have hundreds of microservices, you're telling all of those teams with their own kind of requirements, regulatory compliance and all that kind of stuff that they have to do some throw away code that has to be tested and all of that just for this one period of transition. And so this is why we said, let's take this out of the hands of the developers and have this instead of all these hundreds of people that are working on these applications, put this into the hands of the operators. And so the application developers don't even care. They just say, okay, what do I point my application to? And so that's kind of the rationale here with the assumption that you're moving from one CQL data store to another CQL data store. The other tool in the toolbox here is the thing that I mentioned at the beginning. It's nice to have something that's kind of a dedicated tool and resource to say, I'm gonna migrate and also validate historical data from tables between orange and the target clusters. This is an open source tool again. It does things like that you would hope it does, like auto-detect the schema. Instead of saying, okay, list all the tables here, list all the tables here, it'll do checking on like UDTs, all the exotic kind of like data types and UDTs, collections, counters, make sure that the right time stamps and TTLs are preserved from the origin. So you're not stepping on data when you go to the target cluster. It checks for differences and it even optionally reconciles the data. So if you have some sort of non-item potent operation like an lightweight transaction that depends on the current state of the origin cluster that's not in the state, currently in the state of the target cluster. You can do this after the fact and say, okay, well, was there anything that is off between these two clusters and then reconcile that for me to make sure that both clusters are in the same state? And because you're operating within time windows, you can say, okay, up until like beginning of time to time X for when we started using this proxy, only copy that data initially. And then once you've done that, you have that data done. And then as you move forward, you can say, okay, well, you can do incremental reconciliations along the way to say, let's just make sure that absolutely sure that these two clusters are in sync. I mean, what we've seen is like facility and ease of use and all of that. Those are all great, but consistency of data is what people are very much after. And people don't want to lose their jobs as they migrate from one data store to another. And then obviously you want to be able to tune the reads, for instance, like if you're migrating from one data store to another, the reads on the origin cluster you don't want to interfere with the regular real time traffic from the origin cluster. So this is all kind of like the tool context. And so, go ahead. So Jeremy just works through some of those components. So now we're going to take a closer look at those components that we touch upon. And then we will work through the migration process in a little bit more detail. Next slide. So before the migration, basically your application will just do the regular read and write with the CKL driver against your origin cluster, which is to the right, those orange ring as you see here. And next, please. OK, so this is basically during the migration. So all the activities are having here. And what basically, as you can see from this picture, we have two additional key components into the picture. So that's to avoid introducing any downtime. So one component, as you see in the middle, this DDM proxy. So those DDM proxies basically handle the real time requests generated by the application instance. So the proxy is there to implement the dual write logic so that the same write can be sent concurrently to both origin and target cluster. And then, as you see on the right, so the other component that's called the existing data migrator. So it basically transfer all the existing data from the origin to the target. And these two components, they are basically independently each other. But they are used in conjunction to achieve that their downtime migration. And next, please. So as you see here, we basically set up at least three DDM proxy instances. So they are high throughput and their stateness, which means that they are horizontally scalable. So we configure them to point to both the origin cluster and target cluster with DDM UTL or Kubernetes. And then we update the application client, as you see on the left. So those clients now point to the proxy. So you can actually do a rolling restart of your application servers. So this way, your application can still provide services to the end user without any interruption. And as Jeremy has mentioned previously, the proxy's one job is to keep both clusters in sync per request. And if a write request comes in at a consistent level of the local current, it will execute that write at local current in parallel on both the origin and target cluster. And also, one other key thing to remember here is that the proxy will only return success to the client if both the origin and target cluster have returned acknowledgement that the write is successful. So as we mentioned earlier, we have an existing data migrator, so the component to the very right. So that's basically migrate historical data from the origin to the target cluster. But the existing data migrator can also be served as a validation or reconciliation. And here, through this whole process, as you see in the middle, there's a red arrow that's pointing to the left. That's the read. So in here, we basically just continue to read from the origin cluster until that whole existing data migrator has finished reading all the historical data from your origin cluster and move that into the target cluster. So basically, the target cluster has all the origin cluster's data, and then we can move on to the next stage. So here, you can see that we no longer have the existing data migrator, because there's no use for that anymore. And also, the read arrow to the left is now moved from reading from an origin cluster to the target cluster. So this is where that you are doing the final verification. So basically, you will continue to write to both clusters in case anything go wrong that will give you still a chance to roll back. So if you want to roll back and just point your application back to the origin cluster, nothing is lost at all. But I do want to point out that this is the last step before you're really cutting completely over to the target cluster. So this is the time that you want to do some final validation of the target cluster's capabilities. By capability, what I mean is that your target cluster, there may be a different CKL store. They need to have exactly what you expect from your application in terms of the CKL support and all that stuff. And as well as the capacity, it does have the bandwidth. It's able to process the amount of requests, read and write requests you were expecting from your application. So this is the last stage that you can do that final verification. And next, please. So now, after migration, it's actually quite easy. Basically, now you're fully satisfied that the state of the new cluster so is able to handle the traffic requirement. We'll just repoint your application to the target cluster and the migration is fully done. So next. So yeah, now we're breaking this down into a few high level phases just to explain this a little bit more. So for phase 0, basically, this is the stage that you prepare your target cluster. So this is like you would need to create the target cluster. You need to configure the access to it. You need to create the schema so that the target cluster schema is matching the origin cluster schema. And then also very importantly, you want to make sure you test it. You're sure that your target cluster is working well for the application being migrated. And then you basically just do some validation that this migration is going to work well. This application, once it's pointing to the target, is going to work well. So even if you don't have any data migrated over yet, you just want to make sure that's happening. Jeremy, you want to go to phase 1? So yeah, so that's all the preparatory phase. And then you deploy the DDM proxy through the Ansible automation or through Kubernetes. And then you point your application servers to the proxy. And then it handles the authentication and the traffic routing to the two clusters in each of the areas. So at this point, we move on to phase 2. So this is actually probably the longest running phase in the whole migration because you're basically talking about doing that existing data migration from the origin to the target. And if you have a very large data footprint, this could take days to finish. So the whole thing is existing data migrators job. So yeah, that's basically you just let whatever historical data migration to happen during this time. And people often ask, as we're doing these migrations, how long is this supposed to take? And it's long as long as you want it to take, really. I mean, the data migration is one important element. But as long as you feel comfortable, like being on the two clusters with the proxy, is as long as you're wanting to make sure that you don't have any kind of hiccups along the way, it's really, this is really meant to be a risk of a sort of thing to make sure that you're comfortable with the new cluster and comfortable migrating. And so that leads to number three, is now that you have like a good base of the data sets in both clusters that you think are pretty well good, we've added a feature that's in the middle that says, okay, before I cut read traffic over to the new cluster, I wanna make sure, because we've been bifurcating writes this entire time and your application, when you actually do a cutover, you're gonna kind of unleash all of this read traffic over to the new cluster, right? And so we added a step in the middle that says, you can optionally do this, but you can point still the reads go to the origin cluster, but you mirror the reads to the target cluster and the results are just thrown onto the floor. And the purpose of that is to say, can the target cluster handle gracefully the writes and the reads with sufficient throughput and at those SLAs, like at the latencies that you're expecting? So again, this is another step to avoid the risks of surprises as you migrate to the new cluster. Yeah. So just to add to phase three, so this optional step is async dual read. So basically that means it's not going to increase your latency, like your read is still going from that primary cluster. So yeah, so basically this is really just to make sure that you introduce extra load on your target and your target is able to handle that. And now we move on to phase four. So once the existing data has been fully transferred and any validation check has been successfully executed, target cluster can become your new source of choice for your read. So at this point, you basically flip a switch in the proxy. So then the proxy will basically route all the read to the target cluster. And at the same time, the proxy will still send the writes to both databases so that both databases are kept in sync. And throughout this whole process up to here, as you see at the bottom, we just pointed out that the rollback is still very easy. So if anything goes wrong at any point, you can still just connect your application directly to your origin cluster. And at this point, you don't really have any data loss because it's always be all right. So you don't need to do any restore anything. So it's really like very low risk kind of just rollback. And that gets us to the point of no return, right? So you point your applications directly to the target cluster, everybody goes home, everybody's happy, you're saving money, or you've gone to the new place and things are good. And you still could rollback if you really wanted to, you just do the whole process in reverse. But hopefully this gated process gives you confidence that this new thing is gonna be sufficient for you. And we've had people run on the proxy for a few weeks, we've had people run on the proxy for six months, we've had them run on the proxy during a go live event. And so it's not a problem of running on the proxy, it's just a problem of, or it's not a problem, it's just more of a when you feel comfortable. And the whole thing is just to reduce the risk of going from one thing to the other. And so that's, hopefully gives you a better idea of the high level here. So as we're going forward, I mean, we've tested it in a number of circumstances. We wanna do some more testing with Cassandra 5, we wanna add version 5 support. We have custom authenticator support for I think Instacluster and Ivan. And that's in a branch right now, we need to do a release with that, improve the Kubernetes support to have like some more dashboards and things. One thing it does right now is you can configure it to say the now function in CQL, it produces a time UUID, right? And so if you're sending that to two clusters and you have two different time UUIDs, that's kind of a problem. And so what it does is it says, you can optionally have it parse and rewrite that to say, instead of having now, it replaces that now in there with the time UUID. So it's identical to both clusters. We'd like to maybe expand that in the future, improve the parsing and things because there's a little bit of overhead in there. That's why it's optional to use like date, time, CQL functions and UUID functions to replicate that same pattern. But that's stuff that is in our future work. So before we get to questions, we only have a few minutes. These are some of the links. They're all open source projects. The extensive documentation, it was open source a year ago and we've had a lot of documentation. And Stefano here did a Gitpod interactive scenario. Interactive scenario. So if you're wondering how this whole process works to try it out, you can go to Gitpod. It creates a cluster for you. You do have to create an Astra account because Astra is what you're migrating to for the purposes of this kind of enclosed demo, not demo, but walkthrough. So it does all the steps and you perform the steps and you can see in the terminals what's going on and the application traffic and all that kind of stuff and do the rerouting of the stuff. This QR code is to more easily create an Astra account. It's free. It's like no credit card or anything like that. So this allows you to just kind of go through all the steps which is kind of the outside of the scope of this conversation. But are there any questions? We have some common questions that we have, but any questions? Yeah, in the middle there? You, yeah. Thank you. Yeah, the question is whether or not you use CDC or your internal backups to migrate the data? That's what this handles right here. It's a Spark-based tool that uses CQL to export from the origin. And I mean, there's a number of tools out there, but this one specifically is tailored for CQL workloads to extract and then push to the other data store with the reconciliation and everything. So you can start your historical data migration at any time, even before you even think about putting the proxy in place, because that's gonna take some time. And so you don't have to go strictly according to these phases in terms of migrating the data. You can do that previously. And then only do kind of like the migrate and validate of the data with that Cassandra data migrator of the window of time that you haven't already done. So that reduces the time that you have to have the proxy in place. So you could even have the proxy down to a day or two if you wanted to and done the bulk of the work of migration before that. Is that, and oftentimes people just say, okay, let's start at the time X and then we'll start the data migration. Oftentimes we've seen it where they do the data migration while the proxy's in place. Yeah, the front with the hat. So that's the data migrator will, since Cassandra's kind of like the last right wins, if you take historical data, if this is answering your question, is to say the timestamp of that right is gonna be in the past. And if you get new data that comes in, getting the wrap up signal here, if you get new data that comes in, it'll have a newer timestamp. And so those will reconcile and then the new data will take precedence over the older data. It's very similar. I think this is kind of splitting out applications and I mean, this is more splitting out applications than other things. It solves similar problems, but it has a different, kind of more of a different use case. That's kind of, yeah. Yeah, I was just gonna say, I'm looking up Teotron, yeah. Yeah, that's too interesting. And those guys back there. So, but yeah, that's a good question. Yeah, maybe one more question? Yeah. That is one of the most common questions that we get is that it's not free, right? You have something that goes in the middle, anything you put in the middle is not free. So what we've found through some testing and practical experiences, like a couple of milliseconds. Yeah, so it's a single digit, yeah, single digit, millisecond. Basically the idea is that there's each statement actually. As designed as a very lightweight goal routine. So yeah, it's basically, it doesn't really add too much additional latency, but it's like single digit, like less than 10 or around 10 millisecond with the processing in the middle. Yeah, and that's a per statement. So if you have large batches of many, many statements, the overhead necessarily is the proxy kind of has to go through each statement within the batch and determine if that statement, because you can mix reads and writes, right? So you can have all these statements that you have to kind of say, okay, is this a read or a write, and then I have to send it and that sort of thing. So it's not like it has to do it in serially, but it does have to do the rewrites or the traffic routing of each statement and read each statement. So that's the one caveat, is if you have large batches, then you might want to be careful because the overhead starts to add up. Okay, I think we're done. If you want to talk to us, we're happy to answer additional questions, but I think we have to wrap up there. Thank you everyone. Thank you.