 Hello everyone good afternoon. Thank you for joining for force all the way That's if anybody wants a tip to get a talk in for any summit Just try and put the summit name in the title that typically works for me So my name is Yogi a few things that you will realize from this slide is I'm fond of emojis I Like to travel I've been to all those countries. I'm always exploring ways to get to new countries and Just below that you'll find all my Hobbies I'm Indian. So like cricket comes naturally to me. Yeah, I've been living in Singapore for almost two decades now and currently I'm a solutions engineer at Yuga by DB and Just in case you didn't realize it's not YDB. So it's probably like YDB and Yuga by DB It's a little confusing sometimes, but yeah, we are Yuga by DB I Am working with like our Yuga by DB partners and customers across Asia. So, you know, India Singapore Philippines Indonesia so on and so forth and I have been involved in a lot of cloud native Communities and also app development cloud native platforms in past and now also The QR code is my contact if you want it and those are all the places that I have worked in past two decades, I suppose Why are you here like just in case if you're not sure this is the topic for talking about Scissors and Kafka and everything between I'm gonna have some slide where probably about 150 ish or slides. No, no, no I promise it's going to be only 40, but I'm going to do some live demo. Hopefully it works Did not give my sacrifice this morning. So fingers crossed Um, so A lot of things have happened in last sort of decade Hence that we went through a massive shift in paradigm People are moving towards cloud native applications cloud native platforms. We are talking about Kubernetes Containerization moving to public cloud moving back from from public cloud moving to microservices Moving back from microservices. It's it's all happening, right? I mean It's the sign seems very cyclical and you know, if you were there for the earlier two talks I think the the topic is quite well placed Because it is what what I'm going to talk to you about it very well fits into that same story around DevOps and the consistency so When we talk about Anything in technology today the easiest way to start is if we start with an app, right? which is solving a sort of user problem and Typically when we say an app, it may mean different things for different people If you ask somebody who's born in like last two decades or decade and a half for them an app is iPhone or Android device, right? Pretty much. That's what it is. Maybe iPad, right? Some people like I'm from I'm a kid from 90s. So for 80s or 90s, right? So for me, it's a piece of software, right or a program. So In all in essence, I would say an app is a piece of code, right? That is executable So typically when you take an app, you need to run it now to run it. What do you do? You take a compute, right? The compute can come from a virtual machine or a pod and Your application doesn't live alone. It actually has some supporting things Oh, by the way, one more thing that you might notice about me. I like minions It's competition between me and my daughters. You know, you have pictures earlier. So Typically they have their own supporting services that they require. It could be for some sort of networking capability or some sort of Organization compliance enforcement kind of practices, right? It happens. It is there. And typically in In enterprises, an application is never run as a single thing, right? You normally run multiple instances of it So so far so good Now in in the modern applications in the cloud native world What we have is a container orchestrator or an infrastructure API provider like Kubernetes which basically takes care of taking your code your contentized code your application and Running it on the actual hardware It could be on the cloud. It could be running on premises that that's that's right But again, I mentioned about all sorts of side cars and compliance capabilities and all those things But apart from that, you typically have certain supporting services You may have messaging systems. You may have database systems and everything in between right authentication identity provider all sorts Now as we move towards a more and more cloud native Architecture we try to move as much as possible to the cloud native platform Kubernetes right when I say cloud native platform. It's for me. It's same as Kubernetes now this is like Interesting because you obviously we talk about the cloud native platform and Kubernetes and all but typically you have Multiple sites and multiple regions multiple data centers or it could be cloud regions and things like those right? So everything that we spoke about so far in your mind if you are going through Okay, I can I can actually run this command to deploy this and I can keep a copy of this through this command and blah blah blah Well, all that is doubled now right or tripled in some cases so Something that I always get asked is like what are you talking like running database on Kubernetes? Are you insane or something? Yeah, I am talking about running database on Kubernetes. Why should you actually run database on Kubernetes? right now Kubernetes has actually facilitated Better resource utilization what it does for us well is that it allows you to Use your compute your infrastructure in a more efficient manner So rather than dedicating your compute for certain applications or certain pieces of code You can actually create it more like a kettle Anybody can tell me what's the difference between a kettle and a pet not you So typically for a pet you have a name right it's a goofy or something right, but for a kettle like As a kid when I was living in India. I had like five cows None of them were my pets like I was like okay one of the cows and go milk one of the cows, right? I don't know which one but just go milk So that that's a difference you do not actually have a direct attachment or like sort of Naming or strict dependency on a kettle mindset So that's what Kubernetes is actually providing for us. It's it provides you with the capability to run variety of applications in a very standardized manner and not just running the application But also connecting it to the underlying infrastructure. It could be network. It could be storage. It could be identity provided many things, right? so this actually allows you to make use of the compute for running Multiple databases today if you've worked, I'm sure all of you have worked in some enterprise at some point in your career and If you request for a new database, what's the fastest you've gotten it? If it's not on Kubernetes or it's not on cloud was it Cloud that that's cheating. Let's let's talk about something that that is more traditional I have a giveaway by the way every time somebody says something you you will get a give away What's three? three months that's fast Okay, well the thing is that I have seen like even longer than that but when it comes to Kubernetes you can pretty much spin up databases very quickly and Hopefully if I have enough time, I'll probably try and spin up one for you in front of you. So that is one the second is Dynamic sort of resizing you're able to resize your workload based on your need The demand is higher. You can have more demand is lower. You can reduce it make it smaller and things like those You have an extremely good amount of portability between clouds and on-prem and all that if you run things on Kubernetes, I'm gonna go a little faster because I was there was an issue with the timing. I suppose and More importantly, it provides you with a very good orchestration for your entire infrastructure. So not just You know running your application, but you know connecting to variety of storage and all And of course it Automate a lot of day to work. What is day to work backup recovery restoration upgrades all those is day to work But anything that has benefits has some downside, right? So these are some of the downsides when you are actually running things on Kubernetes you have to be a little aware about stateful nature of your workload because Your your application run and your data may not actually end up on the same node So that's that's a bit of a problem. You can address those through distributed database Persistent especially in case of cases of databases persistent storage is always a problem But that can also be something that can be addressed through a distributed database Another complexity is with the reliable access. You need to have a load balancing to your application to your database Even inside Kubernetes, so obviously if you run it on cloud you have cloud based options if you run it on premises You have to solve for it There is also a big networking complexity, especially when you have multiple clusters If you have two Kubernetes clusters running in two different data centers networking between them is going to be quite challenging Right, so you that is something that you have to be aware of within the cluster if your application and the The database is running on the same cluster. It's pretty pretty pretty good pretty easy So essentially what I'm saying is don't run database on Kubernetes run a distributed SQL database on And that's that is what exactly you go bite is you go bite is a distributed open source transaction oriented database Which is 100% open source. So and it's it's pronounced as you go bite a lot Gotcha's or the limitations that I mentioned earlier can actually be resolved by you go bite in terms of It eliminates some of the need for external load balancers Also, it automates a lot of your day to work that you might be running into plus it gives you the scalability You you have more traffic coming in you can actually increase the number of nodes and If your traffic has tapered out you can decrease the number of nodes you can read and write on all the nodes Let me just repeat You can read and write on all the nodes in Uygur by DB which is a stark contrast to the traditional rdb ms is Without compromising the asset transactions You are able to actually perform the transactional applications that you are you are used to with rdb ms capabilities tables and everything and transactions a Small footprint of you go bite would look something like this you typically have a small management console which basically provides you with the management capabilities and You have database nodes typically minimum three because it's a distributed So three five seven those are all good numbers, but minimum three is needed now what it provides what we have Created is the distributed transaction and storage layer which provides you with automatic sharding and load balancing and On top of that a pluggable query layer that provides you with rdb ms capabilities akin to Postgres and No SQL capabilities which are akin to Cassandra So in a single cluster you are able to perform both kind of data models with Uygur by DB Now what does the deployment of Uygur by it looks like on Kubernetes we use stateful set if you are not familiar with Kubernetes stateful set is a construct in Kubernetes to run Stateful applications. So the ordering of pods and not is taking care automatically by Kubernetes So we use that those we have two sets of pod one is the master which is more of a metadata server master Is a very bad name because the more you say master people go. Oh, there is master So you will copy over know it's metadata server and the T server which is that tablet server The name originates from our sharding name our shards every table that we create we split it into shards Each shard is called a tablet. That's why tablet server and You are able to horizontally or vertically scale each of these components Now I mentioned earlier the time we create a table it gets Into shards called tablets and these tablets are something that are distributed across multiple nodes and All the data that you are storing will actually go Synchronously not a synchronously but synchronously to all the nodes So let me try and show you a very quick demo and I probably would have to sit for this Any questions so far How much time do I have ten minutes, okay? Yes, I am not sure about why DB one thing I remember from my understanding of IDB was it's based on my SQL API and we are based on postgres SQL and I mean in terms of replication capabilities we do synchronous replication of data and Asynchronous like if you are doing it across sites across clusters We can do that Collect your giveaway later. It's here All right So let us look at a database that I have running here. So first of all I mentioned that you go by it is Can everybody see the screen? Is it legible? Okay, so I mentioned that we actually run pods as and stateful sets so this is my database that is running on Kubernetes and you can see I have master and t-server as my Stateful sets and then a bunch of pods which are running here So this is similar to the architecture diagram that I showed you earlier now over here What I can do is I can actually run a sample application that will This is a simple sample application which basically keeps on inserting data into the database It's a workload generator application. So we'll do that now This is like pretty much simple right bread and butter like this is this is okay Now what I'm going to do is I'm going to actually take down one of the nodes of my database and this application should continue to work Without any issues. Yeah, so for that what I will do is I it's it's tough to actually cause a failure in Kubernetes so My take trick is typically I reduce the number of replicas to two like right now I have three replicas. I will reduce it to two and let's see what happens Okay, so I reduce it to two now Something interesting happened. So one of the one of the in-flight transaction Right, which was connecting to that particular node that went down It got disconnected, but rest of the system. It just continued to function now There is a delay of about three seconds between the cluster Rebalancing itself due to the node failure. So it took if you see carefully here It took three seconds for it to just come back to a stable state where it's able to look at each other and We obviously have a small UI to go along with it. And by the way, this is all open source. I'm not using the Enterprise version or anything. I'm just using the open source version This is actually showing you the three master nodes and This is showing you the three tablet servers that we have The one that I have reduced it it is starting to miss the hard beats now because it's down, right? It will wait for this server to come back for about 15 minutes by default If it doesn't come back in about 15 minutes, what it will think of is it will declare it as dead and Any new server that comes up would be bootstrapped for data But in the meantime my application that I was running it just continues to run. No problem without any any downtime I am going to scale this back and What I should see here is Interestingly some of some of the tablets here There are no leaders on this particular node while it was away, right because I mean it is not capable of serving any data What will happen is automatically within few seconds It will rebalance itself and the leadership rule will be restored on the for some of the tablets for some of some of the shards That's that's the thing with live demos they can they can take a while, but yeah there there you see automatically Earlier this number was 0 5 slash 0 now it's 5 slash 1 and it might get rebalanced to maybe 5 slash 2 also So this is how you go by DB because of being a distributed SQL database is able to run in a Cloud data environment like Kubernetes and sustained failures and it can also provide you with the transaction guarantees the asset transactions So that's that let me see Yeah, we saw this particular thing Yeah, just in case if it was not clear, this is like typically let's say if you have four pods which are running A table is automatically split into all these shards and shards are scattered across all these pods The number of copies of the shard depends on the replication factor that you are defining at the beginning of the cluster and One of the shard will become the leader for the shard group So this is how it happens. So all the reads and writes would be have would be done for By the blue tablets, which are basically the leader tablets All the selection and everything is automated Now, I mean obviously if you want to try it out, we have a completely 100% open source version We have Docker container. We are able to even run on ARM machines and all in fact on AWS We are the only certified Distributed database for Graviton. So if that is something that you want to try, you can try that as well We obviously have our enterprise offering as you know, you go by DD anywhere which provides you with a nice portal for day to operation multi cluster management capabilities and all and we have a fully managed cloud offering which you can leverage for your projects You can actually sign up for a pre-cluster today. You can actually get a pre-cluster for life. Just try it out and You can actually arrange for a full blown demo and not like a small three-minute demo just scan the QR code and Fill the form and you get $10 graph voucher. Any questions? Okay, I'll leave it on that and happy to take questions now. Yes, you can use it on there We actually make it very easy for you We have a tool like Voyager completely free open source that you can use for migrating over from my SQL and Postgres and Oracle Yes, third question Okay, good. Good. You will need to refactor your application and there'll be a ETL that would have to happen Good questions. Yes You first. Yes, if you have an application that is currently using Postgres, it should just work In fact, you can use the same Postgres driver. We have a smart driver because we are a cluster, right? So like any other clustered database, there is a there is a concept around it, but you can still use Postgres Driver also Yes, yes Correct. So yes, we've not solved that problem, right? I mean the latency the physics is physics right speed of light the speed of light But the thing is that in case of like asymmetric latency say for example three nodes One is five seconds of five milliseconds away. The other is 10 milliseconds away Then you will effectively be waiting for five milliseconds because we wait for majority of the cluster to get the data So there is that optimization. But yes, we are not immune to underlying network latency We it will become part of your transaction. Yes Good questions. Yeah We are Helm chart is most I would say but we are working on a operator as well We have an operator, but it's not as advanced. Yes. Yes. Yes. So I Can give you like this is a published study on Yugobite website with Teminos, which is a core banking vendor So they have actually managed to get about 450 queries 450,000 queries per second across 39 nodes in three availability zones on AWS It's their Benchmark test and they said it's about 40 percent better than what they had. Yes Your per transaction latency your per transaction latency would be higher because of the synchronous replication Yeah, I mean we what we have what we have observed is Ideally we can bring it down to about 3x of a Postgres database, right But we've even seen like, you know because of the synchronous replication Yeah, because the database database itself the rights on the database are not That I mean there's no latency as such we suffer from the latency on the underlying network if the underlying network has like You know Below millisecond kind of latency, which is like local then you do you should not you should see like at the most about 3x in terms of a transaction But of course we compensated the throughput. Yeah, I mean happy to discuss the use case really Because we have seen we have seen various use cases where We have been able to actually provide better throughput because one of the issues with Postgres You will find is because of a single node master The number of connections, simultaneous connections that you can do to that node is limited so you will be limited by that number but Go horizontally and at least the number of connections. Yes. Yeah, there's no exclusive lock We are a lock-free database first of all second is if your query commit has finished the data has Resiliently stored across the customer Like right very next instance like your commit happened on and obviously at the hood We are using SS tables and mem tables, right and we have a wall log So it works as it goes in the wall log if the node crashed and if it comes back again It will reprocess from the wall log, right if it has not gone through But the point is that if the commit has happened and the control has gone back to the client Then that has resiliently been stored Some other shard which will take up the leadership role will now have your latest commit In fact, I did not mention it in the presentation. We we practically have a zero RPO Like absolutely no data loss in in face of a failure. If you're if your transaction has committed It the data will not be an argue is about three seconds minimal. No, that's for the portal. Yeah The the configuration of the database obviously would I mean that was a minimal configuration I would say but depending on the number of Depending on the number of transactions that you're looking for the amount of workload and all The configuration would be bigger Any more questions? I still have three more things to give away. Please. Yes Yes Exactly Yeah, you do not the application doesn't have to worry about which server do I connect to and No, none of that. I mean being an app developer like that that for me is a Big big not having to bother about which server to connect which cluster to write to and all that none of it Just connect to any one of the nodes write the data there and it will be there. Okay Are we out of time?