 Hello everybody. Good evening. Thanks for coming to this activity. My name is David Perez. I'm from Costa Rica first time in Detroit, so Thank you all So what I'm going to talk today is a little bit about it is cd or it's cd a Couple of concepts that are important to understand the performance of at CD through for the latency how to benchmark at CD with a benchmarking tool that comes with it at CD and some remediation and tips in case we face that problem So first of all, what's at CD? at CD is a distributed Q value store for a store in critical information critical data for a distributed system You can use it for anything, but it's probably most known for being core component of Kubernetes and You don't actually talk to it at CD Right with your applications But that is part of the cube API server functions to talk to at CD to a store and retrieve the cover net as cluster estate So even though you don't directly Speak or talk to the at CD. You can still see issues there These are two examples that I saw One from the at CD log that says that a request took too long to complete and some other Logs in the cube API server saying that there were some Harbids that were missed Indicating that the server is likely Overloaded but what is exactly to be overloaded in at CD terms? so we need to understand two concepts throughput and latency and we understand latency as the Time taken for an operation to be completed and throughput the number of operations that are completed in a period of time So in single terms you can see latency as how fast Operation is completed and throughput how much data or how many operations are completed in that period of time So about the benchmark tool that I was talking about at CD has A known benchmark tool as you're seeing here in the screen that you can actually easily configure you can give a many arguments to the CLI and do different kind of testing against at CD and To to make a quick demonstration here I put on the top side of the screen some requirements and the above it is the Information from at CD documentation they did they did a benchmark on those requirements and I took them But I changed the the disk. I'm using a hard drive disk in the part below And when we do the benchmark we can notice that we don't have that much difference when we are using one connection and one client but when we go distributed with a Thousand of clients and hundreds of connections we start seeing a lot of difference in the throughput They request per second and also in the latency also for the read Information we see a similar thing right similar case that with one single connection. There's not much big deal but when we go a With multiple clients and multiple connections we start seeing problems in throughput and latency Especially on prem, right? So when you face this what what can you do to remediate this? so First of all when when you're designing a system make sure that if you can Design and have ahead on time the number of requests that your application is expecting So do you don't overload at CD? Also try to avoid HDD at all But if you cannot do it because you cannot afford it because the cluster is already there make sure that the HDD is at least 15k rpm and also There's something you can do you can configure with tools like ionize the disk priority So it is it takes priority writing to disks and also turn some at CD settings like the Harbit timeout the election timeout so you don't overload the server and Also, there are some more tools like the this benchmark FIO and also the not conformance from Kubernetes that will help you address more issues rather than just the benchmark for at CD So I went ahead and did a change just changing the HDD to SSD And we can see here the big difference that we have now the same cluster But using solid state drives We can see that the requests per second the throughput goes actually double which is good So Doing a quick recap if you want to know more about benchmarking at the CD especially on prem You can see the operations guide in the documentation and the performance section Also, there's a guest there with some scripts You can use to create your own three member at CD server And it will also give you some commands to do benchmark if you want to test it more. So that's my information Thank you very much. I appreciate it