 Okay, so today's agenda is what is Kafka? We'll just cover basics of Kafka, what exactly it is. Terminologies of Kafka, and the brief about micrography, not in deep, and how we can integrate Kafka with Tante. So what is Kafka? Kafka is a distributed steam processing system. It's a messaging-based system where we can process the data from various systems, and it used on the messaging terminologies. What are the use cases of the Kafka? As you can see, before that, as you can see, there are multiple integrations, multiple interfaces for the Kafka, like consumer and producer, stream processor, and connectors. What are the use cases of Kafka? It, as I mentioned, it's used as a messaging, based on messaging. You can use it as a producer-consumer, where most of the messaging things are used. Stream processor, everything in Kafka is a stream. The stream of characters, stream of binary, is considered as a stream. Event sourcing, event sourcing, you can track your website events in the Kafka. Sorry, website activities, tracking, and event processing quite similar. You can process the events also from different subsystems to different systems in Kafka. Log aggregation, you can use it as a storage for logs, and commit log anyways, it's a messaging, so it's just store the commits, commit logs also. There are four core APIs of Kafka, producer, consumer, stream APIs, and connected APIs. Producer APIs are used to produce the messages to the Kafka, produce the data to the Kafka. The produce data is consumed by the consumer APIs. The stream APIs are usually used to process the streams from one subsystem and one system, and to put it into Kafka, or probably send it to another system for further processing. Connector APIs are basically used to connect the Kafka from a source where you can feed the data into it, like for example, database. You can connect it and then feed the data into that. There are basic Apache Kafka terminologies, which are produced, as we mentioned, producer, consumer, but where they will produce and consume the data from. For that, we need a topic, where the topic will run on. We need a broker, a server, where you can run the topic. The other few, we are going to discuss it in details. The other few are the partitions, offset, and consumer group. As I mentioned, Kafka is a distributed system. It needs to replicate the data across the distributed system. Therefore, we can use the partition. Offset is again a part of partition and consumer group. You can create a group of consumers, which can consume the data from the topic. We will be able to discuss in detail in further. As I mentioned, this is a messaging-based system. So every messaging-based system needs three things. One is the consumer. Another one is consumer, to consume the data. And where to push the data and where from the read the data? We need a topic. The topics here are a bit different from the basic messaging system. There are queues and topics in the messaging system. But here, as there is a concept of multiple producer can produce it on a topic and can be consumed by multiple producers. That's why there is a topic. It's a bit different than the usual messaging terminologies. As you can see in the picture, the multiple producers, like two producers are pushing the data into one topic and consumer is consuming from that particular topic. What they are pushing in? Every message is consumed as a reward for a Kafka and it is pushed into on the topic. Each record is having three main problems. One is each with which it is pushing a topic. A value which we want to consume or which we will be going to use and a timestamp when it is pushing it on the topic. It comes with, it's inevitable. You cannot change it after pushing it on the topic. It comes with a retention limit where you can keep the record on the topic for a certain time. Otherwise, it could cause the high memory consumption over the system and which will desabilize our systems. Every record is pushed with an ID. So you can consume that record with that particular ID. Where to push the record on Kafka topic? As you can see the topic, multiple consumer can push the data on the topic. Each topic can have multiple subscribers. So you can consume the records from same topic. Now you can see here, there are three partitions mentioned. Partitions one and three for a single topic. So what are the partitions? Partitions are, as you can see that, as you know that Kafka is a distributed system. So we can distribute these partitions across the different nodes or different systems. Where you can push the top messages on the topic by mentioning which partition you want to push. So that you know that which consumer is going to consume from which partition. So every record when you push in the topic, it push for a particular offset. Offset is nothing but the record ID on that particular partition. So as you can see it here is zero, one, two, three. These are offsets on a particular partition. So when your consumer is consuming that particular data, you can say that I want to consume this particular data, this particular offset data from that particular partition ID or partition. This facility is provided by Kafka and when you consume that record, it is not getting deleted from that there. It will still be existing on the partition. Who is consuming the record? Again, the Kafka consumer. Well, as I mentioned that, you can ask the particular consumer that which record I want to consume. You can give the offset and you can consume that particular record. At the same time, if you have a group of consumer which can consume a particular type of data from topics, you can create a consumer group and Kafka ensures that the single consumer will within the consumer group only record, and only read that particular record once. The good part about Kafka, we miss one thing. We miss the offset. We also, I also mentioned that every record is stored as an offset into the Kafka topic. But how it is managed, how Kafka knows that I want to create these many replications. Because as it is a forward order program system also, how Kafka ensures that I need to have different partitions on different, how many replications of those partitions I want on different system. Because if it also has a, it is a distributed system. So it also, I'm sorry, for the terminology, failover on any property in it. So we can mention an application factor also about the partition when we are creating. So, for example, if we are giving an application factor of two. So, if I was the different one, it will be a negative twice. One is primary, probably it's another one, secondly. So Kafka is a polyglot. Polyglot, I personally like this feature because I also work on another technology, which is JDG, which is also a polyglot, which provide you a freedom to develop the client for any system in any language. Usually it communicate over TCP, but we can use any language, any protocol like REST, HTTP, to communicate with that particular system and we can use that as a consumer and producer. Now let's discuss a bit about micro-profile. Why we are discussing micro-profile? Because the content which we are going to discuss further is micro-profile compatible. As you can see, the three dark patches, these are the basic micro-profile specifications when it was created, CDI, JSONV, and that size. The other J2A APIs were integrated later. We are going to use CDI to create our client for Kafka with an extension. And how are we going to do that? We will see further. Now what is taunted? Taunted earlier used as a wildfire swarm. It is a micro-profile compatible, as I mentioned. The framework is mainly based on the wildfire application server. And we can create a standalone micro-service-based application out of this. It creates an uber-job, as it is a hollow-job, we're going to use it for running it on Docker. And with the help of that, we can create a micro-service. It doesn't need any application server to deploy the micro-service and we can actually use it as jav-jav and the jav-need. Arogear has an integrated, as it's providing, CDI extension for Kafka, which helps us in developing this code. It removes most of the code which we have to do it and gives us an application which helps us in creating the client for that particular, for creating the client for Kafka. As it has CDI, it's pushed the dependency into the variables and we can use that variable accordingly. If we don't use this particular CDI extension, you can see we need to write this many lines of code and we have to continuously pull that particular topic to read the report and then do the processing of that particular report. But if we use the CDI extension, we can just use the annotation Kafka-config, where we can just specify the server and the code for that particular Kafka broker. And we just need to, as we write a normal, API on any messaging system, on the message, we just need to mention that what is consumer, which topic it's going to read and what would be done. So we can see a small demo, how we can do that. So right now what I'm doing in demo, I am first starting a zookeeper. So what is a zookeeper? Zookeeper helps Kafka in managing the distributed system, over which the Kafka server runs, I just ordered. So when I start this zookeeper server dot start, it start the zookeeper at 2118 over the local and which we need to provide later on. So now we are starting the Kafka server, which is a broker. The Kafka server is created, it's started. Now we are going to create, rather we are going to check the topics which are created over the Kafka zookeeper, not Kafka zookeeper, and check what topics are right now exist. So if we see right now there is only one fruit topic, one topic and nothing else, and one consumer offset. Now I'm going to create a topic called fruit topic and it is created over the same zookeeper. Now when I list it, it is created. Now I already built two jars for consumer and for producer for this Kafka. I just ran it. If you can see all the CDI extensions and everything is inbuilt due to the micro profiles. The throttle is ready, which means the jar is started and our web service is started. Now I am starting the web service for producer, which is running on 811, 8180 and it is running on 8080. So these are my interfaces where I'm pushing the data. On this, I'm pushing the data into the queue or onto the topic and behind the scene, the consumer is consuming the data and pushing the data into a database. After that we'll refresh and fetch the data from that particular database. We push one fruit, you can see it is consumed on the same queue. If you see the logs, it is consumed with ID one. So I am pushing that particular ID from the producer, which is auto-generated on the offset and it is consuming that particular data from or based on that ID. This is what I pushed, this is what I fetched. Similarly, I push another, it is pushed. It is consumed. The two, as I'm pushing it into the database, the two fruits are consumed. So you can see I'm using this Kafka extension. These are the dependencies which are already added from the Thontil and I'm giving this as a CDI extension. Similarly, palm.xml is here. This is when it is fetched, it is selecting that query from the database to show it. This is our get API for the web service. And here I am pushing the data on the fruit topic with an ID and the object which consists of the name of the fruit. So here if you see, okay, I think. Here if you see, I mentioned that this is the local host 9092 is a broker where the Kafka is running and this is the producer. I just need to mention it, what is the key and what is the object it needs to read. I'm just, this is again, a consumer. So I'm just mentioning it in consumer topic. The key type, which is like ID, which is an integer and the group ID, which we are providing. I think that's it from the demo side because we are shooting time. So anybody has any questions? If we are in. Yeah. I saw that you have a consumer and you have a direct identity. Usually you are able to read also other metadata like offset, partition, and this kind of stuff. For example, when you are using it on the screen, I am able to get this data also with the. So if I'm asking you a question correctly, you want to know that you want to see that I can add an offset into that and then I can add data from that particular offset. Yeah, you can read it from the client and they have a place on it, something. Because in Spain, you can get the whole consumer data and you have also the unshown data, but only the key. For example, the topic name, because maybe the partition can be used for some of the running. So that's why I'm asking. So you want to say if I can face the data for that particular topic or for that partition, and then can I face that particular data? So I'm not certain about that. Probably we can, because I haven't tried that. But probably we can. I'm not certain about that. I need to try. I can get back to you on that if you can just share your mail ID. Yeah, sure. Anyone else? Nothing? And these are the references which I followed for this. Thank you.