 So my name is Raghu and I'm going to be talking about ASOP and how you could use it for creating tier data sources. So ASOP is a system that I created its joint work with a number of engineers from Flipkart. So the way we'll probably go around with this talk as first you'll be wondering is why should you even be interested in this? So I'll try to see if you can relate to a number of examples that we'll have in the first few minutes. So once you get interested then we'll see the kind of scenarios that ASOP can fit in and probably how it can be useful for you. Okay, we'll start with this very common question, right? So what data store and the what data store question comes up in a number of places. So we'll let's take a few examples. Okay, how many of you have heard this, okay? Most likely in what situations? Interviews? So when I hear this, I always wonder that folks like Facebook and Twitter probably don't have good engineers. They need to go and find better engineers because they still use SQL or relational systems. So you always have this sort of saying attribute scale to wanting to use a no SQL database. And then when you get to a further choice about the access type, do you want very point lookups? Do you are you're trying to do rain scans? You're trying to have very flexible schemas and then you decide between a number of data stores which is probably relational key value document or let's say columnar. Also when you hit scaling limits of single node databases, you realize that, yeah, all of my data can't fit in a single node database. And then guess what you get to conferences like this and probably a few others and people claim that they have, you know, the best solution for all your problems and just the last talk by Shalem who told you that probably not all of that is necessarily true. Some of it is true, but not all of it. Right. And benchmarks on distributed data stores, rise of memory based and really flash optimized stores. So when you hear about someone saying we are flash optimized and they're talking about using a log structured data storage format. So you need that they're talking. You know that they're talking about something serious because with flash you're there even leveraging. Obviously they're trying to address cases like we are leveling and making sure that they don't have too many of in place updates and cap tradeoffs been discussed a lot in this conference. So you're always trying to trade off consistency availability and probably the, you know, the entire spectrum in between a few other considerations like durability of data, how durable you want the data, distance to memory ratios, how much data can you hold in memory when they say it's super fast, but guess what you have to hold all of that data and memory when you have large data sets that's not affordable to you and you always you're looking for a tradeoff where you can have very large or reasonably larger distance to memory ratios. Right. And and as scale really grows, you're also thinking about size of single cluster that you can can can operate. So all of this is as leading to these kind of questions of what data store should you use and is there a single answer quite quite unlikely. Right. So let's take an example of let's say scaling an e-commerce website and I'm from Flipkart. So let's take the Flipkart example. So this was Flipkart and we can't see too much, but it wasn't nothing great there, but this was in 2007. So however we were, we had a single MySQL instance and we were sending all our reads and writes to this to the same instance. Right. So that wasn't scaling. So we like anyone else, we also moved to doing a reads from a slave and had MySQL replication going. We also had analytic reads going of the same slave. That wasn't working and then therefore we had more slaves. Right. So one dedicated for online reads and another one for analytical reads. So very common pattern. All of us would have seen that. Right. Now in this, what we did, I don't know whether we did it consciously, but over time we realized that we had traded consistency for for availability. Right. And and higher performance. Now let's just fast forward to something like what is Flipkart today. So this is how our persistence looks like. Right. So session user session is a combination of Memcache D. We have Hways assets back end. We have the product or the entire catalog data which is coming from Hways Elastic and and Redis. Cart is MySQL and Mongo notifications is Hways, such as Solar, Neo4j, recommendation is they're generated in Openmar and and and served out of Redis or Aerospy and promotions which is currently getting rolled out as let's say Hways and Redis. So in all of this, a very common pattern that's emerging is there is a source of truth data store and then there is a serving layer that's front ticket. Right. Again, a very common pattern that many of us would have seen. So I'll take another example of because it's tiered data. So we'll take another example of scaling payments. Right. So the Flipkart payments data store has a relational set of tables which stores all of the online interactions with the with the merchant the amount to be charged. So these are just a set of tables like a transaction master, transaction summary, transaction sale detail and so on, which has that it also stores all interactions with the bank. The responses that it has got very, very normalized. We also need a denormalized database where we can use for consoles for handling refunds and so on and why denormalize because the as the size of transaction increases, we can't keep all of the data on the on the online and the live database. Right. And as it grows, we have other use cases data flows. We also have an archive DB, which is for regulatory regulatory requirements. We need to keep a copy of all of those transactions that are there in the online database for for a few years. Right. And we also have another analytics and reporting database which is used again for for doing reports and queries. So clearly in this case, so data needs to flow pretty much on a on an everyday near real time basis continuously and in a very reliable manner. Right. And there are multiple ways that one can think of doing it, which is okay. You write data to okay. Let's take the most simplest way you can do it. Right. So you have the MySQL DB. You can write a nightly cron or a periodically running cron which extracts data. So essentially an ETL kind of a job and then inserts data into your secondary store. Right. Now then you deal with partitions and when I say partitions, not like the Jepsen partition, but MySQL partitions and you keep probably dropping those partitions. Now all the other approaches you always do a dual right where you write into your online database and also write into an archive or a secondary data store. Now any of us here would realize that the problem with that is it's very, very hard to get it right always right because you're talking about rights happening across two threads of executions. You want to have if you want both of them to be transactional then you're talking about distribute and you know distributed transactions and trying to get to face comments and so on. Right. So that's not that's not possible, but if you if you go that way, you're obviously going to miss a few rights. So what happens with this kind of a data consistency in you know, in polyglot persistence? So let's look at some of the caching or serving layer challenges. Interestingly, that happens to be one of the two hardest problems in computer science, right? So what happens with cash invalidation? So why should cash invalidation be such a such a big issue? So let's take a look at some of the caching strategies, right? So you have very often you have a very lazy caching strategy where you have an empty cash which comes up and then as user requests come in, you start lazy loading the cash from the primary data store, right? A very, very common pattern to do that. Now you've loaded data into it. Now what do you do for data freshness? You say that, okay, let me now set ttls on the cash, right? And if you want reasonably consistent or consistency to be catching up reasonably fast, you go and set low ttls. That's fair. And now what happens? Let's take a case where let's say in an e-commerce site, there is a you're doing a flash sale for a product and there is sudden interest in one particular product and all request at landing up, right? So and and take the case where let's say that data is not held in memory. What happens? It exposes. It creates the the thundering herd issue, right? So all requests coming and possibly hitting the same node and all of them finally going and trying to read data from the primary data store and the primary data store, its own vulnerability or around availability gets exposed, right? So if you look at a data store like like MySQL, in most cases, it's a highly consistent data store, but it's not really known for availability, right? So it exposes that construct or that constraint from from the TV, right? So that's a caching problem. Let's say you have a cash size. So if you're if you're talking about a few thousands of items, you can probably size it and fit it into into memory on on a physical machine. But what what do you do if you have about 25 or 30 million items that you're selling in the catalog? You obviously can't fit all of that into memory. So you have a cash size. You're also always trying to to to size your cash in a way that you can you can keep most of the recently accessed data in memory, right? And also distribution and why do you distribute again to scale the number of reads? There's only so much number of reads that you can serve out of a single node, right? So you can probably get up to say 20 K 30 K kind of transactions per second. What happens if it goes beyond that? So it means that you need multiple copies and if you have multiple copies and multiple replicas, it kind of rules out even any write through mechanisms that you can have, right? You can say that oh, I'll have a write behind strategies or how many replicas do you update? And if if you have n number of replicas, you're trying to update it if updates to a few of them fail, then what do you do with the transaction? Do you do you fail the transaction? You're back to the problem of the same thing around distributed transactions, right? So all of this makes even write through mechanisms hard to implement if you're trying to maintain consistency. So bottom line serving layer is eventually consistent at best. So why do we say at best? Because this happens only if if all of your replicas eventually get all of its updates, right? And when you say eventual, there is today there is I've also seen people mention strong eventual consistency eventual consistency itself means that you basically got all of the millions of updates that you're probably making on that on that system, right? You can probably have weak, weak consistency, but eventually consistency means that you yeah, you'd need to ensure that all all replicas get the data. So little bit more on eventual consistency. What it simply says that if you have multiple replicas of data, they basically converge over time. It doesn't time box it or doesn't time bound it in any way. What are the pros so you can scale reads through multiple replicas, right? It gives you higher overall data availability because even if some replicas fail or an entire replica sets fail, you can still serve partial amounts of data. The cons are reads return live data before conversion. So what this means that because the replicas can be at various stages of convergence, they always read return data till what they have really seen at that point in time, right? So it means that they're just returning live data. So it's quite possible that a user goes to a page. He sees a data from one replica, which probably says value X and the next request for some reason because it's all load balance goes to another replica and probably sees another value, right? That's because just simply the replicas haven't received all of the updates. Okay, even achieving eventual consistency is not easy, right? Tripoli tends it requires that at least once delivery guarantees exist and you need to ensure that that happens to updates across all replicas and now depending on how you implement eventual consistency very often a relatively easier approach is is trying to sequence or order the updates rather than trying to deliver out of order messages, right? And when you say out of order messages at the receiving side, the consumer or the receiving replica should exhibit properties or should exhibit commutative properties, right? So it doesn't matter if you're saying an increment decrement or a decrement increment of a value changes in the order of the way they're delivered. You still get the database or the data store to become hold the same value, right? So even just getting at least once delivery guarantee is not it's not easy. So what's ASOP and this is the kind of problem that it aims to solve and again because the examples that I gave you both in terms of the cash update or for that matter the payments database, they are real and present problems that we that we faced and and therefore went and looked at building something which is essentially a change data capture and propagation system, right? So it can detect change and basically propagate changes to the various copies of that data. So what does it look like? So there are three parts. So there's a producer part. There is a relay and there are a set of consumers, right? So producers we currently work with my SQL and HP support both of them. Basically, these are data stores which support some kind of a transaction log, right? A right ahead log of what's getting written. So those right ahead logs, I'll come to that part about how it works get picked up. They get they get played into a relay. Relay is nothing but a memory map file. It's it's a ring buffer containing all of that dates that happened on the respective data store and also maintains subscriptions from multiple consumers, right? And your consumers themselves are subscribers which can be running at very different rates. So they register on the relay. They get the data delivered to them. They transform a map and then you can go and update a number of different destinations, right? So fundamentally, this is it's a very, very I mean at the overall pattern, this looks very much like a pub sub, right? So there are a producer and and and consumers. So what's in this? If you if you notice, there is a thing that we have. We can have data originating in my SQL, but probably eventually delivered to Redis, right? Or it can be something that's coming out of HPase and can be delivered into let's even to Kafka. Okay, so the approach that it takes is is log mining. So is log mining really old wine and new bottle to some extent? Yes, that's because folks like Stonebreaker. We know who is the Turing Award winner and creator of old DB and number of other databases. So he and a couple of others wrote this paper about the architecture of a database in in 2007, a very good read. So I happen to read that. I would encourage all to read and there it mentions that durability is in a database is typically implemented via logging and recovery, right? So that simple log, which we typically we always as application programmers always thought was you write for debug messages. No, in a in a very different world, it's also very different and very important purpose that of data recovery and and persistence. More recently, Jay Krebs, the creator of Kafka famously said that the contents of a DB are just a cache of the latest records in a log. The log is a source of truth and while the database is just a subset of the log. So all of us who are here and I didn't. I did hear mention of Kafka multiple times in this conference and all of us really like life Kafka for various reasons, right? Obviously it's a very reliable and and and a useful piece of software and the creator probably has has you know, has got this right. So what's so special about the right ahead log? What it ensures is it basically ensures that every modification is flushed to disk to disk. So it's not like in the Jepsen test examples that you saw you have in replicas data not getting to getting written to desk and probably losing updates. So with the right ahead logs every modification is flushed to disk and all modifications are maintained in order. So which still means that okay pretty much you can use the use the log. So a shop essentially uses that and you know, in term to just critique the the kind of consistency model that a shop supports. This is a consistency model map from from the Jepsen website. So it's by Kyle Kingsbury. I mean those of you are there with on challenge talk will know. So it basically lays out the various consistency models map that exists and the the strongest being once at the top of the more weaker ones being towards the bottom. Right? So if you say you have a read committed so that is something that which is low on on consistency but high on on availability. A strict consistency like linear is ability for example is too hard and requires coordination across replicas right. So what is linear is ability say it says that if you execute a right the next read across any of the replicas will return the same result. I just think about trying to implement that it's like super hard right. So even even system that probably got close to it like Google spanner right they no wonder they were looking for you know atomic clocks and GPS synchronized clocks and the whole thing to just get the clocks synchronized because it relies on on absolute global time. You take a more less strict one which can if you have in a distributed database you can have replicas which can disagree forever right updates did not reach at all and the worst part is very often you can say yeah I can replay the data but you can replay the data only when you know what you have lost. Typically it happens that you lose a few updates across millions of updates so that's not so bad if you know what you lost and very often you don't know what you have lost and that's where the problem comes right. So how do you know how do you really trust the data store in terms of the replica right. So if you look at sequential consistency and that's highlighted there so all that sequential consistency says is it says nothing about time. There is no reference to the most recent right operation. It just simply says that all of the data that is passed on to the replicas is ordered and it is delivered in the same sequence of occurrence right now an implementation mechanism for this would be basically primary based replication protocols. So essentially what a soft does is basically follows this and when we say that we are weak we can listen into the transaction logs of my SQL or or H base basically what it does is the ASO process runs as a participating replica as a party participating slave in a cluster. So for the my SQL master this looks like a my SQL slave for a H base cluster this looks like a H base region server right. So then you can tap into the whole replication protocol and and basically use that to deliver data. So what about reliable delivery right. So that's where we start the whole premise of the talk was how do you ensure reliability. So if you look at the data store and now that we say that that has a transaction logs everything is flushed to the desk so you see that there are rights whatever 45678910 happening in order and there in the log so in the producer so the ASO producer is basically a slave or replica of the same transaction log so the data is being delivered to it. It does a transformation and stores it into the into the ring ring buffer and at the same time it's also checkpointing the offset in the in the replication log in the bin log. So in case of my SQL what what we do is basically that offset or the checkpoint we call in this it's called a system change number which is a monotonically increasing number which is nothing but a long computed from the the bin log number and the offset of the record inside the bin log. Okay so that checkpoint is stored either in a file system or in zookeeper will come to where else we use zookeeper similarly on the subscribers end to they're receiving various number of updates all of them store again store similar check points in in a same or the similar data store. So those of us who have used let's say Kafka know that we the clients are supposed to hold on to offsets right store offsets and then go and resume from from those offsets. This is very very similar right. It's just the offset computing mechanism changes between data store so for my SQL again like we said it's the file number and the offset we also implemented an se generator which can take care of master ship transfer changes right. So if you have if you promote a hot standby to a master then obviously the bin log offset may not necessarily remain the same right. So we have implemented something that can take care of such transfer scenarios and in case of H base there's nothing like an offset inside the the wallets but we computed based on the timestamp of the walleted and some more seeding of data to make it monotonically increasing right. So they see and compute basically differ differs between these different data stores and subscribers of course after they write to the destination store they can go and store the offset and that's how it maintains reliable delivery and what this helps is even if the the points of failure right if the relay were to go down if the subscriber were to go down it doesn't matter because when they come they always resume from that offset. Okay so with that you can kill nodes you can bring them back on we are not gone and run tests like the Jepsen test because none of this if you see the destination data stores they are as consistent or the quality that they exhibit exhibit is left to that data store right. So if you say elastic search loses data and certain scenarios it will continue to lose but what is up just ensures is it ensures that the data that was there from a source system is delivered to a destination and that is based on these offset. So even if you go went and you know mugged around with IP tables or if you went and changed the timestamp the SCN is not derived from that right. So if you went and of course went and changed the offset in the DB and then you went in that I don't think is a network partition kind of a simulation right that's like someone really seriously trying to mess around with it right and and what this and and if you see the the guarantee tries to ensure there is an at least once delivery guarantee so which means that the destination data stores also exhibit some characteristics of item potency. Right so the destination so what we have done in many of our even when we went into production in some of these systems if we for any reason think that when there was a bug and data was not delivered we just simply reset the offset to probably a day prior and just rerun the whole set of data right. So that replay ability is important and and probably the characteristic of it being that you can replay from a specific point in time is important okay. So rather than in this case try to solve for saying the system will not fail you'd rather deal with saying that what happens when the system fails and how can you recover from that right and and I think that is a is a more practical approach in our when you're working with this distributed systems. So on the event consumption side this is all some sugar coating that we do you if you think that you don't want to write code and you want to consume data from my sequel and sale right to elastic search or right to Kafka or let's say H base right you don't need to write code you can just write mapping configurations the underlying libraries map it and an update your destination store right. So you can basically define it's a it's a kind of a simple mapping where you say what is the source what is the destination you can apply some transformations you can do some grouping I'm not I won't go into too much of specific but yeah you can do all of that you can also do multiple mappings like payments data store for example we use the same stream to write both into our denormalize my sequel instance as well as to H base it's it's the same relate just that the consumers are to separate separately map consumers. On the consumption we also support clustering now you might wonder why you think that a producer is the one that needs clustering so why we're doing clustering on the consumer side if you really look at it from a workload characteristic the producer is just simply tailing the right ahead log right so if you really see the speed at which it can operate even a single threaded process can operate at basically this transfer speeds right what you'll have as a bottleneck is the destination data store where you're trying to write right so on the consumer side we support partitioning and we also support partition failover through this clustering mechanism which uses again zookeeper in the back end so those of us who know we use zookeeper know how good it is with the respect to you know holding on to these basically these these replicas and partitions right. And even when you have partitions for example you can I mean you might wonder as to okay there are n number of consumers so there is an ordered set of events have happening on the producer what happens to the same ordering is it maintained when it comes to any of these consumers right you don't want to have in a case where updates on the same entity gets delivered to multiple consumers and then you have out of order rights right the we use a mechanism of of basically hashing in such a way that all updates to any entity always gets delivered to the same consumer so sequencing is maintained that way right. So any of the system of this kind of it can't work if you don't have console so the the most useful and the very basic console that we have here I mean and I basically master some of the IP address because this was a production console can't see much I'm sorry about that but what it shows here is basically shows a sale snail graph where it shows the producer the top most is a producer and the bottom are all the set of consumers so if someone your PSE or you know or PSNES are looking at the at the network so at at monitoring this system they need to raise an alert if if they start seeing divergence and this is just simply tracking the SCNs so the producer SCNs and the consumer change number so it's basically saying how well the consumers are catching up you see a single line that's because it's all just overlaid on on all of them right so it's like one producer and about six consumers all of them getting overly and then there you can see partition wise you can see how each of your consumers are doing which partition is is is a hide or which partition is behind and and so on okay going on to summarize what it is a swap is basically a pops up like change capture and and propagation system what it is supports multiple data stores so we already support my sequel and and H base on the source side and my sequel H base elastic search and Kafka on the on the destination side it suitably abstracted you can keep adding more destinations as you please we also use support radius on the destination some performance so benchmarks typically you say lies but okay this is a production load so I can say with reasonable confidence we ran this was a data migration thing that we did when we moved the payments data store from just being my sequel into that tiered to my sequel and and H base store so we were moving data from from the my sequel instance to H base so the data size was like 800 gigabytes right so those of us here who operate my sequel instances you know that your ops guys will totally disown you if you say that you have databases bigger than 500 500 GB or so right so are this database had grown up to 800 GB and we had to do something about it so a relay running on a single Excel VM 8 cores and 32 GB we had four consumers because I said consumer is where you'll have the bottleneck with about 50 200 partitions we achieved a throughput of 30 K inserts per second right so the data transfer from so if you see it as a single threaded read of the my sequel bin log which really is not where the bottleneck is the bottleneck was even with 200 partitions so the right was really on the on the consumer side but with something like H base being I think this right we got was very high for the kind of a data store because H base itself is an up and only data store right so it's extremely fast with rights so with that we we achieved 30 K inserts and I think this data of 800 GB I think got migrated over a few days I think two three days or a period of time the data got got moved what isn't it isn't it's it's not an exactly once delivery system it is not a storage system by itself because barring that memory map file that it is holding on it's not holding on to any any more data so that memory map file we have gone to to reasonably large sizes which is I think up to 64 gigs of storing avro serialized data which is reasonably good for a few thousands of hundreds of thousands of events but not for holding millions of data there is no global ordering across different data stores right so if there are n number of replicas so it is not responsible for trying to make sure that all of them are reflecting the same copy of data so which then gets into the territory of linearizability which clearly it isn't there okay so the project itself it's open source it was built as open source continues to be remain open source you can find it on on GitHub there is a support group you can write to the users so all engineers who contribute are right here in Bangalore so it's quite likely that you'll get a response from one of them or from me so you can write to us we have multiple production deployments at Flipkart so just to tell you so our entire payments database so it's super critical obviously right payments so it uses ASOP our wish list system so we had a use case where those of you have seen wish list in Flipkart today you can't share the wish list you can't tag it you can't allow search of wish list data and so on the primary data store is my sequel but we the when you read the serving layer is from elastic search right so that is live so you can and the intention really is tomorrow you can you can make it highly accessible through various free text search on top of the wish list and so on you're trying to discover oh that's my friend like this this kind of a thing and does he have us in his wish list so you can you can execute such queries and so on we also have used it for a complete rewrite of our promotions data store which is running of H base and and ready so that's going live pretty soon we also have a data platform so the data platform team that you saw there and and in any data platform when you talk about say dimensions and facts so dimension dimensions in our case are essentially entities so you can say user is a entity and we have events right so events are browsed by at the cart and so on so the events are very voluminous they get ingested from logs and all other mechanisms but all these entity updates and when you're getting it from logs obviously there can be some loss right so the logs getting written and so on but the entity updates we want them to be very very reliable you can't afford to lose any of those updates and all of the entity updates currently for all user data from in Flipkart is going into the analytical data store through ASOP I've I've come across even couple of interesting use cases where one team was using it for doing a live data migration so they created a new schema so on the new schema the schema change users are accessing it they can't bring down the system so they keep the same data store active data is getting migrated and at the same time the new database has gone live right so multiple of these kind of use cases we have started using them and it's it's only it's only growing so whatever you see on GitHub so the way we work all come it's going to GitHub they get pulled and then get deployed into our production setup so whatever you're seeing in GitHub is always ahead of the version that we have running in production okay so basically that it's not like we we develop it now and then try to push it out so the related work so we use the whole idea of the producer relay came from a database which LinkedIn created we looked at it we liked it what they did but we weren't very happy about some of the stuff that was very closely tied and in terms of how it could be managed so we took out portions from that when I say portions components from it and and libraries use it within within a shop and it's still embedded we also use other for my SQL we use the my the open replicator the Google project but we forked it for in order to support my SQL 5.6 which was a version that we are using we also use for H base replication we use a project called side effect processor of those of you don't so we use a number of other open source libraries in this as well very interestingly as of May 2015 so there was a paper which came out from from Facebook on a system called wormhole and while reading through it I just got to know that the first interaction itself said we'll have this problem of in their graph search if you're searching for one of your friends who used to live in a state X and then moved into state Y when I say state as an you know geographic state and it takes a while before those changes propagate into this cash layer it can be returning inconsistent data they were okay with that but what they wanted was something that could reliably send this data to to all of their caches so reading through so it was it was great because reading through the paper so they haven't they haven't done anything about open sourcing the system but the paper was out there and when we read through it we saw that there was so much in commonality with what we were trying to do right including how partitions are there how ordering is maintained they up they apply or use the same kind of a log mining approach and and so on so this is again related work and what's good is even if they don't open source it the paper has given us ideas to further enhance and probably build in this system so that make it essentially accessible both for our own teams as well as others who might be interested couple of references the paper that I said about architecture of a database so great read so many of us think I mean ever wondered how on database works I would say go read that it's not like it's not extremely technical but very useful in terms of design and the the worm worm hold paper itself you can go read through that and and probably then you know compare it with what we do at a SOP and you know you can draw your own conclusions right so that's all I have so this project like I said is open source used in production so if you guys are interested go have a look license we so all of the flip card open source projects and I can speak because I lead our OSS initiatives we have chosen the patchy to license so you want to go and build a closer solution you're more than welcome to do it or use it any questions so a SOP is basically a pub sub system right it's a pub sub like system because yeah the closest analogy you can see is it's like it's a pub sub system okay so I'm just wondering if there's any back pressure so if any of the consumers are so if it's writing to any of let's say it's writing to Kafka and let's say that there's a network partition so does it affect the relay in any sense no so what happens is given that it's a ring buffer so the events the most recent events are what are kept in the in the relay right so if the we have something called a consumer falling off the relay so the scenarios like you said right so consumers not able to catch up so when the consumer falls off the relay so there is something also called as a boot strapping data store which is nothing but another consumer which is keeping track of all of the data so when a consumer comes back up it can consume from the catch up boot strapping data store and then go go into online consumption we have done that but what we found more useful also we have we have implemented another one called a blocking bootstrap so which basically allows for first time cold starting of consumers and these kind of scenarios like replaying replaying the stream you mean when you say cold start you mean replaying the story this is sometimes suppose you want to just like how we did the whole data migration from things so we have seen that the sometimes the producer is so fast ahead of the consumer that it basically all consumers fall off but they're not able to catch up so we even have a blocking bootstrap so you can you can use that as a bootstrapping mechanism thank you hey yeah as I said the a shop producer is actually single threaded as it is the replication is single threaded it reads from the bin locks it is single threaded man yes can it be extended to multi threaded like today like I said you it is work you can you can do there's nothing stopping in the producer because every producer is it's written independently you can go on you can go and write it okay it is doable but then if you're doing concurrent read so you have to do some kind of partitioning across bin lock saying what producer ordering between the partitions and so it you can maintain ordering within the partitions so you can have them push to multiple to multiple relays so you can do that and how do you see we solve the challenges of partitioning and the offset offsets in the partition like many threads reading tailing oh no I see it will be very hard if you're trying to read from the same bin log if you're trying to read using multiple consumers so which really means that you have multiple think of it as you have multiple my sequel slaves right and then you're trying to coordinate sequencing across multiple my sequel slaves but if you see as you said key the bottleneck is not at the producer but it's actually at the consumer yeah so you don't probably need to do it at the producer at all if you're seeing 10 K transaction in a second then you see the bottleneck actually at the producer not at the consumer because it is single threaded and descending one by one we had 30 K I was telling you about 30 K inserts right so 30 K insert so we were still bottleneck on the consumer not on the producer and this was on a on a on a on a VM so if you had probably run this on a bare metal just a producer on a bare metal you'll probably be able to get to much higher transactions probably 30 40 K kind of inserts per second thank you