 All right, why don't we get started now then, yeah. So welcome. Hope you've been enjoyed the summit. Yeah, yes, no? Yeah, great. All right. So I work for Waxknob, so I thought I'd put the front page title, the sales guy here would approve. I'm the techie, so there we go. So I'm just gonna do a quick introduction about myself. So who am I? Self-engineer, DevOps, architects. Been a Cassandra user since 2009. So I was one of the early adopters of Cassandra when I was working for Sky. Yeah, so I think that was the CTO at the time came to us and said, hey, you need to be active, active with your database. And we're like, you wanna pay for some Oracle money? And I thought, right, okay, let's start looking at this no SQL, but we didn't need to call them no SQL at the time. Cassandra, MongoDB was coming out at the time and we were testing them. We'd created kind of a multi DC environment using desktop machines in the office and plugging cables in and out and to testing but I was just gobsmacked as to how well Cassandra performed then. So I just kind of fell in love with the technology. So then I joined Datastacks in 2013, left in 2016, set up my own company called Digitalis and then finally now Axanops. So Axanops is a Cassandra management tool that we built. So go check it out. Anyway, so I'm a, what am I gonna talk about? So I'm a frugal person. I take great pride in eliminating waste wherever possible. I think it came from the times when I was little and my mum used to tell me like it takes 88 days to grow rice and I shouldn't waste any and kind of had a profound effect on me. And so any bits and bytes and CPU cycles, I hate waste, okay? It's just kind of I tell you something about myself. My first job was at Citrix and I was working on the Citrix protocol where you could have the full desktop experience at 28.8 kilobit modem. And that kind of appealed to my sense of frugality at the time. Okay, so I'm a few years down the line, things like XML became prominent protocol. And when I looked at it, I was like, oh my God, what is this? Right? And you know, it was meant to solve the problem of machine structured language over the wire as well as being human readable. And I don't think it achieved either of them. So anyway, that's a little bit of a background about myself. And I've got a guest speaker here today, Joaquin Casaras. Do you wanna tell us a bit about yourself? Sure, thank you very much. Yes, so my name is Joaquin Casaras. I typically stutter especially on my name, so please bear with me. Yeah, I'm a software engineer. I typically find myself in the infrastructure space just because I move in the startup sort of realm and DevOps is kind of hard to find. So I do a lot of infrastructure stuff, a lot of performance. I am data stacks employee number nine. That was a very long time ago since then, been at the last pickle, been at the Cassandra space, helping Umbull improve their workloads. They had hour long batches, dropped those down to 18 minutes for them. But yeah, so currently I'm a Cassandra consultant, still in the Cassandra space. Yeah, so if you have any questions, don't hesitate to find me in the hallway and back to Heather. All right. So what we're gonna be talking about today. So you saw the titles, great, you Cassandra metrics the right way and the dirty way. Now, I kind of had to choose a title that's appeal to the sense of humor of Patrick McFadden who was doing the selection of talks. So I kind of came up with a title but I'm gonna be talking about kind of something a little bit boring to this conference. Everyone's talking about AI and generative AI and things like that, really exciting stuff and I'm gonna be talking something a little bit more boring but hopefully you'll get something out of it. All right, so Cassandra metrics. So you guys here probably because you've probably monitoring Cassandra at some point and collecting the Cassandra metrics. Those of you who haven't, so Cassandra exposes metrics through the JMX ports through the M beans and it's out of the box it generates over 20,000 metrics or well maybe even more these days, right? In Cassandra 4.1, Cassandra 5.0 they're producing a lot more metrics now and that's a lot of metrics that's it exposes and every table you add to Cassandra database it starts generating even more metrics and after about a couple of hundred tables you create starts generating one over 100,000 metrics and that's a lot to collect. Now those of you who have been in the realm of operations and collecting metrics for Grafana for many years fairly kind of standard way of monitoring Cassandra some of you maybe using Datadog or other metrics collection mechanisms but a lot of people have been using the standard JMX exporter. Yeah, and that is kind of the open source way of collection of metrics in your Cassandra database or other Java systems, right? And there are a number of other kind of Cassandra specific metrics collection mechanisms like MCAC from data stacks, Instacluster created one as well and Creteo it's a French kind of ad tech company. Now these are fairly kind of standard common ones that we see in the market. A lot of them expose the metrics through the open metrics API. So what that means is something like Prometheus has to kind of fetch the metrics against the Cassandra database. Now metrics collection can be expensive. I don't know if you guys have noticed but when you collect a large volume of metrics from Cassandra you get quite a big CPU spike sometimes. And then occasionally if you got a lot of metrics to collect it takes several seconds to collect per scrape. Now collection of metrics from the database and having an impact material impact of CPU is not great, right? So this is why I thought to talk about this. Now at Axanops we've, as I said, I'm a frugal kind of guy when I started looking at this CPU cycles being wasted from correction of the metrics I was like, okay, gotta do something about that. So we started looking into the JMX exporter at the time. This was about six, seven years ago now as to why it was causing such a CPU spike on every single collection of metrics. So at each metrics collection iteration so when something like Prometheus hits that open metrics port, right? What it, so I was following the trace code it actually, it creates, it does query the list of metrics through the JMX port, right? So it gets the whole big list of MBs, okay? So it's a big iterative kind of a for loop gets all the MBs and it creates the MBs objects and then filters out the ones you don't want. So you can actually filter out, you know the metrics MBs you don't want. So that's kind of an expensive iteration and it does that for every single collection now. So generating objects, you know when you've got hundreds of thousands of metrics that's a really expensive thing to do. And then the next thing it does is, so as I said it's a selection of metrics, sorry about I went a little bit too far in the previous slide. So it selects the ones you want, okay? And then starts invoking the scraping process. And then after that each MBing has a number of attributes. I don't know if you looked into the MBs inside Cassandra but for example, the table metrics you might have a throughput and then or latencies. And they have like 99th percentile max and minimum and various percentiles, right? So the number of attributes in each MB. So you have to iterate over those. So it creates a list of attributes and then it has to iterate over those attributes. So number of iterations going on here, very large volume of a list of MBs and then each MB you got attributes, so iterating over those and then it tries to get the value. Now that kind of a process beam back value kind of method. These code by the way is just a small snippets. It's not the full code from JMX exporter. So I'm not showing the entire code. It just doesn't fit in the screen. But yes, it's a very big expensive method that gets called on trying to retrieve the attribute value. And then after that Java com Sanjay Max Interceptor default MBing server interceptor Java. Every time you try to retrieve the value it has to go through this permission check code inside the JDK, right? And this also we found that, you know causes a number of CPU cycles to be consumed. So as you can imagine, if you're doing this for like 150,000 metrics it's going to consume a lot of CPU cycles, okay? And then after that you finally get your value. Very, very expensive thing to do. And most enterprises because of the expense lot of them are kind of collecting the metrics at every 30 seconds, every minute. That's a fairly standard kind of a collection cycle, right? Or frequency. So I wanted more insight into Cassandra. More high resolution metrics and we decided to do something about this. So how did we make it more efficient? Now those kind of huge iterations, yeah, querying the list of MBs and querying the list of attributes on the EGM beam. We decided that's probably the one of the most expensive part of the collection, right? So we decided to build a Java agent that like the one that JMX exporter provides or the MCAC, et cetera. Using ByBuddy. So I don't know if anyone's used ByBuddy but it's a bytecode interceptor that you can manipulate the code dynamically inside your Java code, right? Now we use ByBuddy to create a Java agent. And then what we did with that is that the metrics collection, we didn't want to create a big list of MBs objects every single iteration. So we pre-created that. Right, then we have the list of objects that's been pre-created for the list of MBs in the Java agent that we built. So we didn't need to go through an expensive process of creating those objects on every single iteration. That saves an enormous amount of CPU cycles. Okay. The next thing we had to do is then once we got, we create the objects and the attributes list created, then iterate over them to collect the metrics, right? And we are doing collection of metrics every five seconds. And it's not that expensive to go over the pre-generated objects. So then we also iterate over the MB attributes that's also pre-created and then we get the attribute value. Now, because of the bytecode interceptor, right? We did a really dirty hack and this is why I call the title dirty. We did the dirty hack of bypassing the permission check in the JDK. Really naughty, but it is effective in saving more CPU cycles, okay? So what that means is our agent is so efficient at collection of metrics that you barely kind of see the performance impact of the collection of metrics from Cassandra, especially when you got thousands, well, well over a thousand tables, okay? All right. So just to validate how performant this is, we wanted to do a bit of a benchmarking comparison of the kind of impact the metrics collection has. And we kind of contacted our old friend, Joaquin Casares, who is gonna give us a bit of information about his test he's run done and the results. So Joaquin. Thank you, thank you. Yeah, so we went through, we ran this test on Cassandra 4.1, which was the latest release of the time that we made sure it was a valid test, had all the new features. Some features are a little bit, cost a little bit in performance. So we wanna make sure that this is as up to date as it was at the time. We're using open JDK 11.02. We ran this on M7I large clusters. And for each of these tests, we did have three independent clusters, all running the exact same way. The only difference was we installed different agents. For one agent, we did, or for one cluster, we installed the JMX exporter in agent mode to make it very similar to the other two tests. MCAC was already running the agent mode and we had the AXNOP agent. These metrics were also collected every five seconds, which is a little bit different than Prometheus, which only collects every 30 seconds by default. And we wanna see the load that the number of tables would create on our metrics gathering. Because every table that gets created comes with additional metrics and these metrics are hundreds of metrics, so the metrics pull grows really quickly. So in our first start here, when I was trying was, I figured, hey, let me start with 100 tables and see what sort of load we got. And as you can see, AXNOPs was our CPU spikes were fairly low. We saw very quickly that the other two were pretty high. I went in and ran that test for 200 nodes, or sorry, for 200 tables, and we see the same sort of results. At 300, we couldn't run any more tests on JMX exporter and I'll show you that slide in just a little bit. And just for fun, I figured, let me run AXNOPs at 1,000 tables, see what sort of stats I get. And even then, we're running at about 33% a CPU spike. So this isn't 33% all the time, it just spikes up to 33. So when you look at the CPU load average over the last 15 minutes, that's when you see why I ended up stopping that test on the JMX exporter pretty early on because we already have a very long queue of Cassandra processes that need to be processing, that can't be processed because we're spending all that time on metrics. And formally, during my time in support, I would see times where we would have to turn off our metrics, turn off our logging in order to have that cluster stay on. And that is a little bit counterintuitive because if there's an issue with the cluster, I need full visibility, not running blind so it could stay up. So yeah, MCAC lasted a little bit longer, but as you can tell, it wasn't the same sort of lightweight model that the AXNOPs agent had. And at 1,000 tables, we're running at 0.33% of CPU load average over 15 minutes, so it's super, super light. So in order to do probably one of the wonkiest comparisons I've ever done, I continued on with that and these charts show AXNOPs table, sorry, the AXNOPs cluster still had those 1,000 tables. I went in and dropped the tables for the MCAC cluster and the JMX exporter to only 50 tables, so that way it was a little bit more fair because I wouldn't want to run a performance test on a cluster that's already under provisioned. So as you can see here, even at 1,000 tables, AXNOPs is still reading a lot faster, it's still writing a lot faster too. And in my mind, these numbers, like they're all kind of relative, it's on an M7I large, so at the end of the day, what I'm really looking for is how fast that job complete. So even with the AXNOPs agent monitoring 1,000 tables, we were still able to complete 1 million rows, writes and reads at half the time as the JMX exporter. The MCAC exporter is a little bit better because it does do caching in the back, but we're still having to take a little bit of that hit because we have another caching layer that's kind of affecting our processes. And another thing that AXNOPs was really proud of is the ability to have everything be, like Kyle also said, no waste. So I went ahead and looked on the wire to see what sort of information was being transferred off the machine. So even though MCAC and the JMX exporter are only exporting metrics, and AXNOPs is also exporting logs at that time during all these writes, you can see that we had 700 megabytes over the course of 24 hours being exported out to AXNOPs to servers. MCAC, on the other hand, had 186 gigs being exported of just metrics. And that's because a collective framework doesn't provide compression out of the box, so there's no way to turn that on. And that's the sort of difference you're getting. MCAC was a little bit better, but we're still over the course of one day transferring five gigs. So when you're looking at what sort of capacity your Cassandra cluster can be doing, with the AXNOPs agent, we're really running super, super slim, almost no impact to your cluster, but you're still getting five second granularity on your data. Yeah, so I think we worked on the network protocol side of things very hard as well to try and minimize the bandwidth consumption. As you know, a lot of public cloud vendors like to charge full bandwidth utilization, especially cross AZ charges, NATs charges, et cetera. We wanted to minimize the money that goes to Mr. Bezos's pocket. So I worked very hard to make that as minimum as possible. Hopefully we don't have any AWS people in the audience. Okay, good, good. So just kind of final things really. So both MCAC and JMX exporters model is to open a port on the Cassandra servers so that Prometheus can scrape. I don't really like that model of opening a port on a database so that metrics can be collected. I prefer for the metrics to go outbound from Cassandra servers or database servers in general, just an architectural consideration. Simply because if you configure your Prometheus wrongly, you badly, things can go wrong, it could absolutely hammer your metrics collection as well. So I just think that's not a great design in general. So we actually kind of go outbounds rather than sort of opening up for collection purposes. As I said, network bandwidth utilization is a big consideration. And also the frequency of the collection. So we were talking about most enterprises collecting the metrics every 30 seconds of every minute. And simply because it causes quite a bit of impact in terms of the collection or the network bandwidth utilizations. What that means is you see the chart at the bottom. This is a chart of a five minute window. So if you collect that every 30 seconds, you see the chart that is on the left hand side, right? And what we do is collection of the metrics every five seconds. So you get a much more kind of clarity. It's like putting my glasses on and being able to actually see what's going on with your servers, even at these small time range, right? So that kind of gives me a good insight into how Cassandra behaves because sometimes the clients come in and creates a spike. But if you're only collecting every 30 seconds, those spikes get smoothed out. The way you do the throughput is that Cassandra is incrementing the read counts and then you are applying a rate function against the counts on the timestamps, right? So then if you're kind of collecting every 30 seconds, then those spikes disappear in the charts. So this is why I honestly believe that collection at much higher intervals is important. And that's why we've implemented. So I think my clock is telling me 29 minutes, 33 seconds. So it's time, but thank you and do take a look at AXNOPs. And yeah, if you've got any questions, I'm happy to answer them. I've got my email address on there and you've got some QR codes for your AXNOP links. So thank you. Questions? Creation of MBs. So objects creation is quite expensive in Java. So when you go hundreds of thousands, yes, it's a very expensive process on every iteration of metrics collection. So yes, that's a big one. But I keep pursuing for more CPU cycles saves. It's one of them law of the whatever returns it's called. More time you spend, less you get it back, but I just pushed it as far as possible. Yeah, diminishing returns, that's the one. Yes, the security sort of things. So in terms of the jumping over that security, it's because we're kind of intercepting the Java bytecode and it is a very well-known tested code. So, and nobody has access to our code to be able to invoke anything. So we determined that was safe enough. Well, it's just a permissions check. So it's not like an authentication as such. I think if you're hitting the JMX port with a username and password type of things, it then checks that. But because we got a Java agent, there is no kind of authentication anyway. So that's what it is really, yeah. Yes, so we've been doing this for Kafka as well, exactly the same. So when you got lots and lots of topics, it generates enormous volume of metrics and we found exactly the same, even though it's in Scala and stuff, but yes, it works on Kafka. Yes, how do we make it so small in terms of the bandwidth? Let's just say we're not using ASCII text like open metrics where you have enormous text describing each metric. Okay, hopefully that makes sense. All good, excellent. Well, thank you for coming.