 engineer in Red Hat currently on the division project and I'm going to take this role from being a shadow format unit or division. So we will have a brief look at what change data capture is, then of course we will focus on the division project and from there because we are supposed to focus on non-arrangement databases we will talk about changing the capture for MongoDB specifically and after that we will actually have a look at one of the possible use cases for change data capture time in MongoDB. At the end there will be place for questions and hopefully some answers. So what it is, change data capture. I very much like this definition that change data capture is a process of recognizing changes in a data system and they're delivered to some downpour consumers so that these consumers can take action to base on these changes. This diagram actually portrays one of the variations of change data capture based on mining of the transaction log. It's somebody here who doesn't know what a transaction log is. Just for the remote audience, the transaction log is sort of a canonical source of truth for each databases and that's the thing which actually records all the operation of an old transaction which then can be used for example to recover something nasty happens to the database for example some analytics or anything like that. So we have a database and we have some users which are actually executing all your regular create update delete operations and modifying your data. Then we have a CDC platform which is reading the database transaction log and it's emitting events about these changes to any possible downstream consumers which can then for example store your events as data in some data lake or data warehouse or perform some calculation and statistics on top of that. Beside this transaction log mining approach there are some alternatives mainly we can implement a CDC based on queries and based on database triggers. However these approaches do have some disadvantages for query based change data capture but one of the obvious disadvantages is that you have to constantly execute your query so that puts additional strain on your database and you are also not able to actually capture all the changes because how do you actually write a query which captures deleted data? The answer is you don't, you can't. With triggers you can actually recognize all the changes to your data but you get a very database specific implementation and you are also limited by the expressiveness of the procedural language which is supported by your database so you might have actually issues getting the changes out of the database. The most common use cases for change data capture include but are not limited to things like the data application or actually cache invalidation of search and index updates. You can also build auditing on top of it and last but not least you can actually use CDC with microservices to implement some of the architectural patterns which we don't have a look at at the end of the presentation. One of these things is like reliable data exchange between microservices. It brings us to the division project so what is division? Division is actually an open source complex change data capture platform which covers multiple databases. We have quite active and large community because the project has been around for a while and it's been used in some relatively large scale deployments by our community already. When I said that we support multiple databases we have to talk about what called the division connectors and these connectors can be actually divided into three different groups so to speak. First there are the core connectors. These are the connectors which are the source of these connectors is actually part of the main division repository under the division organization and these connectors are developed, maintained and supported by the core division team. Then we have community connectors again these connectors are housed under the division organization umbrella and give-up but they live in separate repositories and they are actually developed by our community contributors. So for this we have DB2, Cassandra, Wittes and our newest addition with Cloud Spanner. The third group which for me is probably the most valuable one are independent connectors. These are connectors built by completely third parties and different companies outside of the division team and Red Hat and everything. These are built on top of our command CDC connector framework so to me these are the showcases that the project kind of took a life on its own and it's now spread beyond the boundaries of our core team and Red Hat in general. So how can you run these connectors? First and foremost our database connectors are actually an implementation of Kafka source connectors so do we have somebody who doesn't have any experience with Kafka? Okay so Kafka is let's say event messaging platform and there is this component called Kafka Connect which provides the WIT to run services or connectors which can get data either into Kafka or out of the Kafka. The ones which are getting data into Kafka are called source connectors. So that being Kafka Connect connector it means that it requires Kafka and Kafka Connect to run and the events are then stored in Kafka topic. So you can deploy DB2 as Kafka Connector on top of Kubernetes using the StreamZero operator, hierarchy, you are just in time. The other option is actually embed DBZero into your application through something called DBZero Engine. DBZero Engine actually started as a tested dependency because we needed means to actually prove that the connectors work but as it's usually with good testing tools they kind of became used in production by some of our community members so we decided to improve it and it actually built another runtime around this DBZero Engine and that's how DBZero Server was born. DBZero Server pretty much fulfills the same role as Kafka Connect does but it's simpler, more lightweight runtime and it's really meant for different data-sync architectures where you either don't want to or can't use Apache Kafka. We also support many other things besides actually being able to store these events in Kafka. You can get them into Amazon Kinesis, Google Pasta or simply send them over HTTP. The new addition to this ecosystem is the DBZero operator, which was just released in the first preview version and you can actually use that to deploy easily in DBZero Server on top of Kubernetes as well. Now to the third important part of this talk, we also need to say what the MongoDB is because that's going to be the database of folks for today. MongoDB is a document-oriented, no SQL database, no SQL database that means it doesn't deal with relations. Similarly, I support them from my SQL boot and document it means that the data is actually changing or stored in form of documents. Other than that, there's a pretty similar structure to the database so you have the database itself. A collection would be an alternative to what the table is in a rational database and a document would be counterpart to a row in that table. The document delsubs are stored in a format for JSON, which is a binary representation of JSON. However, it's not just binary JSON, JSON goes beyond what a regular JSON is. For example, there are additional data types which are supported in JSON if you wouldn't find in JSON. And MongoDB also has native support for change of the data observation. Before we actually get to extracting the data, we need to talk about how the database can be run. First and foremost, there is this standard of deployment but that's not really meant for production. It's recommended just for testing and development environments. And since there isn't any way to actually extract data changes from standard of deployments, we don't care about it and we cannot possibly cover it. So, the database is two, a replica set which is the basic unit of MongoDB deployment. It provides some basic data identity and HA features. And then if you want to actually have a horizontal scaling of your data, you can use MongoDB Sharded cluster deployments. The primary one is a replica set, as I already said, and it's a typical primary secondary topology where you can actually see that we have a client and, first of all, all the drives are always directed to primary which holds the data and the data is replicated to secondaries which express as a beta replica. And second, they can actually take some of the load from primary to support the reads from the secondaries. As a basic topology, if primary goes down, then the secondaries are actually entering an election phase and they will, among themselves, elect a new primary. This is done by the fact that there is a heartbeat going on between all the replica set nodes, which means that every node in a replica set is actually aware of the existence of all the others. The data application part is actually the core of what we need to know for the ECM. You can see here, we can see some data copying going on, and this is so-called operations log. At the beginning, I've talked about databases having this transactional log, and this is actually an equivalent in MongoDB for this. So what is an operation log or a book for shorting MongoDB? It's a cart collection holding a rolling record of data modifications. What does that mean? Well, first, as I said, what a collection is, is an active element to what a table would be in a relational database. And what it means that is a cart collection, so it means that there are some restrictions on top of it, and obliques specifically can be restricted in two ways. First, there is a maximum configurable size of all the data within the oblock, and there is also a configurable time retention in hours. If the oblock either grows very large outside of the maximum size, or a record is in oblock for longer than is the maximum retention time, then entries will start dropping out of the oblock, and these actually mean that the data will be lost. So long story short, oblock actually keeps all the changes down to the database for a specific period of time. So depending on the configuration of the size and the retention of oblock, that's how far back in time you can actually go to replicate all the changes which are done through primary. And this oblock is actually the means how data replication is achieved in a very classic deployment for MongoDB. So if the secondary two replica went down, and the primary had an oblock which is configured to retain data for two hours, for example, then as long as the secondary would be back up in less than two hours, it could just really stream all the required changes. If the secondary was done for longer than two hours, then it would have to discard all its data and actually copy everything from the beginning. For sharded clusters, it's where it starts getting complicated. We have to first add a component called mongo, which is the mongo router. It decides what shard you want to go, and then we have separate shards. But first, what actually shard achieve is a means to horizontally scale your data. So it means that if you have, for example, a custom recollection of all recollections, then a portion of the data is actually distributed to each of your shards. And the mongo router is the only part which actually knows about all the specific shards. So that's sort of the glue layer on top of regular replica sets, which decides where should, where is rerouted, or how to manage set results or perform sorting across these shards. Because the shards themselves are individual replica sets with individual operation logs. So they are not the variable each other's existence compared to genomes inside a replica set. Getting the changes out of this oblock can be tricky sometimes. Since version 4, MongoDB actually provides an abstraction called change-streams. This is a feature where you can actually subscribe or open a change-stream. Again, the database itself will push information about these changes in form of changes in documents to the client. This is the feature which DBZM actually employs to extract these changes from MongoDB. You can actually start the change-stream at some operational time, which again can go as far back as the maximum retention of the oblock. Because change-streams are still just an abstraction on top of oblock. Or when you actually get some change-stream document from this change-stream, you can use its ID as the resume token to start receiving these changes after a certain operation from the past. How does the entire change-exduction process function with DBZM? Well, first in DBZM, we have these two phases. First, the phase is something we call snapshot. That is what, at the beginning, we actually query and transfer all your data into events to actually get the starting point. Before we do that, the first thing we do is we actually obtain a resume token. This is the point of time where we started the connectors. From there, we will actually stream all the changes. But before we start streaming, we transfer all the currently persisted data into events. Once this initial data copy phase is performed, then we switch into what we call a streaming phase. That's where we use this formally obtained resume token and start streaming the data. With each change document we receive from the open change stream. We do two things. First, we actually transfer it into events and we deliver it into the thing, which could be Kafka, Redis, HTTP, Google Popsup, or any other messaging system we support. We also store the resume token for this particular change into our offset storage so that in case there is an outage, for example, there is a connection loss to the database, we can resume streaming from that point of time. Once again, I mentioned change streams being an abstraction on top of operational logs, so that means this offset can be at most as old as is the maximum retention time for change streams. In case you actually, for example, have an outlock which retains changes for two hours and the connector is down for more than two hours, then when you restart the connector, depending on configuration, the entire process will happen from the beginning and you will again have to undergo the initial copy of the data, because otherwise you would be risking data loss and this is something we cannot guarantee. We cannot guarantee. We don't want to guarantee that we don't lose events. It's a bit more complicated, but we get to, then we get to sharding, because in this case, we have two options of actually extract the data. In both cases, we are opening change streams, but in the so-called replica set collection mode, what we do is actually in order to achieve higher throughput and higher performance, we open an individual connection to each of the shards within the sharded cluster. This has the advantage of having larger throughput, because we stream changes down to each of these shards separately, but it also means that we can never actually guarantee, for example, the order of changes performed across these shards. We guarantee it always if there is a new entry, which we are observing, we can guarantee that all the changes will be in the right order for one specific entry, so we will always get creation first, then there will be modifications in the right order, and if you delete it, it will be always at the end. Is there a question in the background? So we can guarantee the order of these changes for one particular document, but we cannot guarantee the order across documents, because there is no means of actually knowing for the using whether a change in shard 2 occurred before or after a change in the shard 1. We can have these guarantees in the other connection mode for sharded cluster elections, called sharded. In this case, we open the change streams again to the Mongol router, which glues everything together, because the Mongol router in turn opens change streams to each of the other shards, and it performs a synchronization, and it orders everything as it should be, but the price for it is that there is some overhead, and then there is some delay before we actually can get these changes from all of those shards. This is all a theory, but let's us look at how we could actually approach some practical problems using division and Mongol Ruby. So imagine we have a simple order sender base, which needs to do two things. First, we need to store an order, and we also need to send a message to another service called shipment service, and we need to do all of that reliably. What's the problem? Well, dual write are prone to inconsistencies, so if you want to guarantee that we do both of these things, what's the first solution which comes to your mind? Any ideas? That's the same time. Not at the same time, but sort of both or none. You have to guarantee that both succeed. Distributed transaction and two-phase commit would be an option, but you may not want to use distributed transaction, and for example, if the system where you want to send the message is, for example, Apache Kafka, it may not even support it. So how do you actually ensure that if you store the data in the database, then the message is always delivered as well? The solution is something called a works pattern, and it starts with actually eliminating one of these external systems, and since the database belongs to the service, the one system we eliminate from this storage process is sending the message, or eliminate, we move it somewhere else. So instead of initially sending the message, we save the message into an outbox collection together with the original order, and we do both of these inserts in a single document transaction. That means that the eliminate the service coupling, but then again we still haven't sent the message to the Apache Kafka, so we need to introduce sort of a work, a work process which would be responsible for the delivery of these messages. The way it will look like is that we will have this order service that stores both the order and the message, and then we need something which will read these messages and guarantee that they get delivered into Apache Kafka. So what's the process for the work I think is to actually send the message? First it needs to pull the collection and check if there are any unprocessed messages, so it gets a message, then it needs to send the message, and then it needs to mark this specific message in this outbox collection as a process. What are the messages of such approach? Yes, you cannot guarantee that you actually do this near real time, and then you again are pulling the collection so you have a similar problem as you would have with the query-based change data capture approach, and you can't really solve the fact that in order to reliably mark it as process, you would have to again do both of these in transaction, the sending of the message and marking out process. This is something which cannot be solved, so the best we can do is actually have at least one guarantee. We can guarantee that each message will be processed, but we can't guarantee that it will be processed more than once. And last by moment at least, if you are the developer who is supposed to implement this pattern, then your files are boilerplate. You've gotten from sending a message to implement an entire worker process which has to clear a database, send message, and ensure that it does it reliably. That seems like a lot of work. However, this is actually where Divisium can help you because it gets rid of the entire need to implement this worker process responsible for getting rid of these messages. You just plug the Divisium to your database and it will completely act as the as the worker process. It will be observing any changes by opening the change streams to this alubox table and it will be editing these messages to the desired Kafka topic. There are several mandatory but configurable part which these messages need to get. So we need something called an aggregate type. This is just to distinguish different message types and we need some ID of the related object to this message and then you need to provide some payload which is any data or none data which you wish to send with this message. Then there are additional fields which you can add as a wish. For example, in this case I have a field called type which distinguishes the type of the event. Let's demonstrate with a simple demo. So first of all, I have a local Kubernetes cluster running where I have few things. First, I have Kafka deployed through the string operator. I have some tooling image which provides this thing type. Kafka can actually have a look inside the Kafka topic and I have Mongo database running and I also have a database operator. The first thing I'm going to actually do is deploy the division and before that happens we can actually have a look at what is the configuration. So this is the configuration for division and we can see three important parts. First, we have this same configuration portion which says that we will be actually sending these events to Apache Kafka. If we wanted to, we could replace this Kafka configuration. For example, Google pops up and the messages will be edited to Google pops up. Then we have this source configuration which just unsurprisingly contributes to MongoDB connector because we want to extract these changes from MongoDB and the most important part for us is actually this transform part because the Outlook's pattern support is implemented as data transformation. Again, really simple, it just says that we want to use this Mongo event router. This is a configuration to which topic we want to actually send these messages to. In our case, it will be events.gov and then a name which we will configure as the aggregated type. Then this additional configuration just says where the type field should be placed and then we kind of want to unwrap our data of our data to actually get rid of something that big is a format envelope which we are using for other messages. So in the meantime, it looks like GB zoom actually started. So first, we will actually simulate what a service would do with the data. So we will open a Mongo shell connection and we will just demonstrate that there is no Outlook's collection yet. It is not there. I have a simple JavaScript code. We will do a simple simulation. First of all, we will open a MongoDB session and then in the transaction, we will create a new order and we will also create the message which will just contain this order as its payload and we will store both of these in transaction. So now we've started and we can have a look into Kafka and see whether we've actually received these messages. I will just connect to Kafka. I want this mode. I want it unbuffered and I want a topic. I want it to go from the beginning and I want a topic named events order and just to have it look nice, we will plug it through. So we can see that we've received this message which in this case we are able to show just the payload but if we looked at the entire Kafka message including the message envelope then we would get a slightly different output and in this case you can see that in headers we also have this order created value for the type field. If we were to cancel the same order, again I will cheat a bit and put it from here, then you can see that we actually received another message and in this case the payload is empty because we haven't stored anything. In this case we were just canceling the order. So the takeaway is from this is that CDC is quite useful to event driven architectures. DBCM is actually a project which provides the need to employ chain data capture and others pattern provides efficient means for microservices to actually rely on exchange data and it's an alternative to a distributed transactions and true face comments. If you have some questions I will try to answer them. Still at least one is delivered, right? Yes. Anything else? Specifically with Kafka and it's something Vojta here would know more about there is now a possibility to have exact ones, right? But that's specific for Kafka at this moment because Kafka is the one doing the deduplication. If there are no more questions, thank you for your attention.