 Yeah, so, yeah, hello everybody, so we can get started. So this is a presentation about CDC event-driven cloud-native application with OpenShift. I'm a senior consultant at Red Hat. So CDC, CDC is a design pattern used for event-driven cloud-native app, and a lot of time we use it with OpenShift. You can continuously identify and capture incremental data changes. And you can see this happen in real-time data replication across many databases and replication through transaction logs. It trigger event-based data changes and capture and populate the event to microservices within your system. So CDC is based on upstream DBCM project. It works with Spring Boot application and integrates with OpenShift. Why do we want to use CDC, right? CDC increase in data collection, right? A lot of time, query takes time, especially when you have a large database with growing sizes and growing data in the database. CDC will basically only focus on the change, right? You look at the data, you look at the data change and act on the data specifically. So that means you don't need to query the entire database. And this is super fast and has a lot of performance improvement. You don't need to do a full DB import and export, right? That takes time. And again, you know, this is a real-time data replication and that will become possible by looking at just the data. So change data capture. When we integrate as Spring Boot application running DBCM and monitor the transaction logs. In this case, I'm going to show you an example how to monitor transaction log through the table called a student table in a Postgres database. Then we have a Spring Boot application configured with DBCM connector that will invoke a listener method when the record got updated in the student table, the invoker will notify the listener. So this method will sync the database in the student index on the elastic search. So this is a high-level architecture about what we could do with CDC. On the left side, you have Postgres database. You have different transaction coming in. You could insert a record into the student table. You could update a student record. You could delete a student record. And the Postgres database is using port 5, 4, 3, 2. And every time we have a transaction log from the student table, it got sent to the CDC relay, which is a DBCM object. And this is using port 8080. And then at the end, we used the DBCM to populate the update, insert, delete into elastic search. In elastic search, you have a student index that basically allow you to index and search for the specific changes through the transaction log. So the Postgres database, as you see in the architecture diagram, it is using port 5, 4, 3, 2. Elastic search on port 9, 2, 0, 0, HTTP and 9, 3, 0, 0, transport. In OpenShift, we use a template to create Postgres database and elastic search instances. It's super simple. It's just a 2-yaml file, and you could create it to objects. DBCM and Postgres image allow the extraction of changes committed to the transaction logs. And this is a foundation of CDC. If you look at the Postgres deploy template, this is one of the tools for creating the deployment for Postgres. And as you see, we specify the name as Postgres. We specify the container, the environment name, the database called StudentDB. So hello, Ape, I guess we are losing the connection with you and you are not. Can you hear me? Yes, a lot. Yeah, now we can hear you. Perfect, perfect. Yeah, so I'm going to go over the slide. Can you see my screen? Yes. Perfect. Yeah, so I was talking about the elastic search deployment template. This is where we specify the template, the deployment definition for elastic search. This is getting the image from Docker elastic.co for the elastic search version 6.8. We specify the port number 9200 and 9300, 9200 for HTTP, 9300 for transport. That's the template. And then we have the elastic search surface template. We specify the kind as a surface. All you need is specify the port number 9200 and 9300 and the selector to be elastic search. So now you have the four EMO file definition from the last four screen. All you need is to go to ocrpi-f, apply the change for elastic search deployment, elastic search surface, post-quest deployment, post-quest surface. Now you've got four objects created. So the next step, you could look at the post-quest student table. This is the definition of the student table. You have the ID field, which is the primary key. You have the address of the student, the email, and the name of each student. And then after the configuration, you could start cloning the project. The project is in my Git repository. It's a Java Springboard application. After you've cloned the project, you could open the pom.xml file and look at the EMO dependency. You could see that we are using DBCM as a dependency. We have the DBCM embedded and also the DBCM connector post-quest. So these are the two objects we need for this project. And then you have the CDC listener. The CDC listener, the class constructor load the configuration and set a callback method for the handle event. The handle event method is invoked when a transaction is performed. The embedded engine class is a web for the connector that manage the connector lifecycle. And this is the constructor that we talked about in the last screen. The CDC listener basically take a student connector, take a student surface, and then inside the constructor, it create using the embedded engine from DBCM. It create and configure the engine. The engine is using student connector, and then it has a notify method to notify the handle event when a transaction is happening. So this is the configuration for the embedded engine and then we set the student surface to the student surface of the class. So the start method is called when the DBCM engine is initialized and started asynchronously using the executor. And then the end method is called when the connector is being destroyed. This will stop the DBCM and merging the executor. The handle event method is invoked when a transaction is performed in the student table. The handle event method will also identify what operation took place and call the student surface to perform the create update, delete on the elastic search. So for the private executor, this is how we create executor using the DBCM object, executor.news and go thread executor. And then once you have the executor, you have two method, stop and stop method. The start method you would call the executor to execute the engine. The stop method will check if the engine is not known, we're going to stop the engine. And then the handle method we talked about in the last screen, it takes the source record. And then using the source record value, it will check based on the operation. If it's not read, if it's read, we don't care. It's not making any changes. It is not read, right? We will go through and create a message map. If this is a delete. Hello, we are not able to hear you. So I guess we have some internet. Can you hear me? Yeah, it's fine now. Yeah, so it was an internet connection issue. Can you see my screen? Yes. Perfect, yeah. So going back to the slide, the student connector listen to the change from the student table using the Postgres connector class from the DPCM object. Offset storage, keep track of how much data has been processed from the transaction log. This is helpful when you have database connectivity issue. The code can go back to the offset storage and keep track of how much data has been processed to pick up the next data that needs to be processed when the connection come back. Yeah, exactly. It could resume from the last three-day appointment and it will occur. The file offset back in store, classes used to store the offset in the local file. So since this is in a local file, if you don't have a connection issue, if you have a connection problem, it still can read the information for the file offset. The connector will record the offset in the file for the new change. DPCM engine will flush the offset based on the flush interval. Once the data has been processed, it could flush every five minutes and delete the offset. So the DPCM connector configuration class, you have a student connector method that would return the DPCM configuration and the configuration is created based on the connector name, offset storage, offset storage file name, the flush interval in this case is flush every 60 minutes. The name, the post-quest connector, the database server name, host name, pod user, password, DP name and so on. So all these different configuration parameters were part of the DPCM configuration. From the student surface, the main, there's a method called maintain read model method that handle update, insert and delete. The student repository is an interface that performs the quad operation on the database. So the maintain read model basically create a map and then create a student object or convert the student object from the student data into a student class. And then you check, if the operator is deleted, then it's going to go using the student repository which is an interface to delete the student by ID. Otherwise, if this is a save, we're going to call the student repository to save the student object. So now we have the different pieces in the Java code. The next step is to look at the deployment to OpenShift. When we deploy this class, I usually call OCE new app. You created your project for OCE new project, whatever project you want to call it. And then inside the project, you call OCE new app. You specify the URL. You specify the context directory. This is a Java app. You can use a CDC while fly 13. And then you will do and deploy the apps for you in OpenShift. Once the deployment is complete, you could do OCE get surface. Then you should be able to see the three different surfaces. Now you have a surface for CDC which is coming from the Git repository, the Java code. You have the Elasticsearch. You have the Postgres database surface. So the next step is to expose each one of them. You can call the OCE exposed surface CDC, exposed surface on Elasticsearch and also exposed surface on Postgres. So that you can call the wow from external. After that, all you need is to call the OCE get wow. And it will show you the wow for the free services. So Postgres database transaction. When the query can do an insert, update and delete record from the student table. So each query will trigger a CDC event on the Elasticsearch. So these are the simple query that we could use for testing. Insert into our student table with the ID, name, address, email, specify, inside the value. Update students, set the email to a specific email address based on the ID equals one. Or you can delete the student based on the student ID. So now you have the application deploy up to OpenShift. You could do some testing. So what you could do, you. Hello, we are not able to hear you. Can you hear me? Yeah, no. It must be an internet issue. Yeah, no problem. Can you see my screen? Yes. Okay. Yes. So, so we were talking about how to test the CDC. So for the insert scenario, you go into the Postgres instant, one, the insert query into the Postgres database. And then after that, you can immediately do a curl command called the Elasticsearch endpoint and make sure that the Elasticsearch will return the object that you just created, like ID one with the name, with the address and the email address. So that's the insert scenario. The update scenario is also similarly, you can connect to the Postgres database one, the update query. And then you could do a curl command on the Elasticsearch. And this time you see that the email get updated based on the upgrade query to Jill at gmail.com. The delete scenario says similarly, you go to a Postgres database, delete the student where ID equals one. Right. And after that, you do a curl command on the Elasticsearch instance. Then you will see ID one had no name, address and email after the record got deleted. So, yeah, in conclusion, you see from the presentation CDC work really well with Cloud Native OpenShift and then enable a very easy, highly distributed event-driven, transactional log-driven, stateful microservices deployment. Right. And the event stream created based on log-level change. All the application listen to event and perform action based on the data change. You can see for, based on, you know, insert, update, delete or read, you can do different action. Provide consistency and also portability using the BCM connector. And you can use the BCM connector to monitor record-level change in the database and publish the change to streaming surface such as Kafka or AMQ. Right. So, you can extend the implementation to even more distributed system using Kafka and AMQ. Distributed Kafka connector cluster also provide high availability, scalability and fault tolerance to improve the performance of your application and system. Right. The architecture is open source project and work well with any programming languages or development framework. So, yeah, that's a high-level CDC introduction with OpenShift. So, if you guys want to play around with it, get the slide from the Git repository that I posted on the chat and then you can start deploying that into your OpenShift instance and start doing some more customization. Right. So, this is the link to the CDC project. Cool. So, that's all I want to talk about. Yeah. Thank you so much. Thank you so much. That was great presentation. I love it. So, if anybody have any questions, they can unmute themselves and ask a question over here. Okay. Any question? Any people who want to ask questions? Okay, yeah, if no more questions, you can always bring me offline. I'm more than happy to help out with any CDC projects. Thank you. Yeah. Thank you so much. Thanks again.