 My name is Hugo. I'm part of the record team. I'm Mexican, Massachusetts in the United States. I'm based in the Westford office and I'm a specialist in APIs and messaging and also an open source advocate. And when we used to travel, a fan of traveling all around and then so good man. So I really like to try new food and street food and so on. So I really miss being in the Chagrabub like sometimes we were able to taste really good food there. So for today, we're going to be talking a little bit more about Kafka. We will be introducing some challenges and interconnection of systems. And we will talk about the Apache camera project for those that haven't really heard about that one for the integration. It's a very interesting project. And then we will be talking about one of the sub projects within the Apache camera umbrella that is the camera Kafka connector project. And why is it offering to us at all? So I hope you like the session and please remember you can put some questions, comments in the chat and we can follow up the conversation using discourse as part of the event chat. Okay. So first thing, let's talk about Apache Kafka. So Apache Kafka is the cookie is the center of this of this chat. And let's get a review of what is Apache Kafka. So remember Apache Kafka was a project originally created around 2010 within LinkedIn and it was mainly focused on trying to track all the different web clicks that you do in the site to give some analytics and more information and insights on how the site was being used. And for that, they were trying to get all this information coming from the web system and being able to easily move that information that around and being able to store them quickly because the amount of events that they have in the site was growing so much. They were facing some limitations on the traditional messaging infrastructure to record all those events. So that's why they decided to focus there for some trading. This publish and subscribe messaging system that allows you to implement a very efficient broker that can handle thousands of messages and events per second. And it become and it has been growing into becoming a data streaming platform. But in the core of the system, it is distributed commit log. So it is a set of events coming into a log file that is just appending. So it's very focused on having this high throughput nature. And as a result, it has a distribution in nature because it's doing sharding of the information and these files across a pool of broker notes that allows you to have a very good throughput. However, most of the people that are starting to talk or getting into the Kafka world, they are focusing on just implementing Kafka by itself and thinking that perhaps that implementation, it's enough to solve many different problems and use cases and technical patterns from microservices or even driven architecture. So let's think about this of a Jedi journey, right? And one of the things that I tell people is when you are focusing on Kafka, it's not like you are in the, you know, the last Jedi type of movie where there's only one Jedi and they only need Kafka for that and that will solve, you know, all the universe problems. You are not alone. When we're talking about Apache Kafka, it's more when we're talking about the first movie where there's a lot of Jedi around. So don't think about Kafka by a goal itself, but more like Kafka as a tool for that. And this is because Kafka, it's a broader ecosystem, right, beyond the broker. So, you know, there's the Apache Kafka project under the Apache umbrella, but that does only include certain features like the APIs and some projects like NeuroMaker and so on. However, you know that we need to be able to connect with other systems. We need to be able to interact with other applications. And even for the LinkedIn use case for metrics and analytics, you know, you cannot access a Kafka UI and being able to just review the topics and so on. It's not part of the original project. You need to connect with other different parts and systems to be able to export that information to showcase those data visualizations or even able to provide more data into your system. So let's talk about how we can then be able to connect with Kafka and integrate our systems. And one of those mechanisms within the same Kafka ecosystem is the Kafka Connect API. So the idea of the Kafka Connect API is to have this wrapper around the consumer and the producer APIs. And it's this framework that allows you to easily compose transferring of data between your Kafka broker cluster and other data systems. And for this, it offers a series of mechanisms and components that facilitate data conversion, scaling, and load balancing because you can create a Kafka Connect cluster that is able to handle the lifecycle of your connectors or your connector plugins. That's how it's calling the Kafka Connect world for those pieces of software that allows you to have specialized connectivity with systems. And it defines a framework for having sync connectors and source connectors depending on if you are moving your data from or of Kafka. And also remember that the Apache project only includes two connectors, the file sync and the file source. That means that the ecosystem around more connectors needs to outgrow from the Apache project and it's being provided by additional providers. There's other companies and providers that have developed their own additional plugins. And you will see that the databases have their own plugins or other Kafka providers providing enterprise support. They also have their connectors. So this is an ecosystem that's growing and growing and growing. And let's review why Kafka Connect. As I was mentioning before, it's part of Kafka itself, right? So you are talking about it appears that it's very well crafted within the ecosystem to match the capabilities, use the same versions and so on. And as I mentioned, if you are using the cluster nature of the Kafka Connect, if you are using the distributed one, it's scalable and distributed by default. You can also rely on storing your tracking information for the offsets within Kafka itself. So it's automatically able to outscale. And also it offers some things like additional transformations. So transformations, when you are moving your data from one source to another, remember most of the times you will want to have some data management so you can, you know, remove fields or do some minor changes. So this framework is not an integration tool, but it offers something like the single message transformation that allows you to do some kind of these activities for easy of use, right? And allows you to have this streaming integration and batch integration. So you will have your connectivity easy and configuration based. And you can also write your own connectors. So in this case, remember that when you're, we're talking about this positioning on the first movies of the Star Wars trilogy when there's a lot of GLIs already available. Sometimes you need to remember that most of those GLIs are coming from different types of species and they talk different languages. And this is very similar to what we have with Kafka, right? So we have Kafka and then we need to connect to very different type of systems from coming from Slack to MongoDB to passing through Telegram or exporting those to Elasticsearch. So sometimes those systems are having protocols that are not included in some of the other connectors providers. So you have to either ask the system provider to give you an easy to consume API or you need to craft and develop your own specific connector. And sometimes that is very complicated. It's like if you need to understand how to connect to each one of those services being just the Kafka add-in. So taking the same approach as we were mentioning on the Star Wars trilogy, why don't you benefit of some mechanisms that are already available for you like in this case C3PO who is very fluent in over more than six million forms of communication and you're able to use and rely on these different type of technologies to be able to help you with that integration within Kafka and other systems. So this is the approach to, you know, reuse some of the content that is already there that has been proved for a lot of time because projects that have been dealing with integration since a while and then being able to tune your Kafka ecosystem to be able to use this kind of components. So let's get a little bit more into what it's Apache Camel. So remember Apache Camel, it's another project under the Apache Foundation that helps us to deal with integrations. So in a nutshell, what is Apache Camel? Remember this open source project that is basically the Swiss Army knife for framework as for integration. So it has a lot of components, more than 350 of them to be able to do data transformation, to do protocols and connectivity as well as the implementation of enterprise integration patterns. So you can model your connection flow or your data flow as a route and then you can do interesting things like being able to do aggregation or splitting or content enrichment and so on. And this is a project that has been around for more than 10 years. It's a very active project and the community is growing and blossoming all the time. They're moving according to the new technology and they are evolving all the time. So if you want to take a look closer on how Camel routes look, you can see the following slide where you define using the camel DSLs of the domain-specific languages to define how your flow or your integration is going to be described. Okay, and I'm sorry about that. I was mentioning how the Camel routes look like. So it's the way to check this as a DSL where you can put like from where is your data coming and where is the endpoint that you are sending your information to. So we have different ways to express those DSLs. They have the Java version that it's based on Java language where you have a fluent DSL that helps you to map this kind of approach or you have also the XML DSL where you can define a spring kind of objects in XML and then define again using tags from where your data is coming. In this case, for example, it's coming from a file that it's stored in the data inbox folder and then it's going to be sent into a GMS queue that it's called order. So you can see there's using this URI pattern where you define what is the component that you're going to use and where is the destination for that. And the camel architecture is a little bit more complex. So if you want to take a look, it's like if you really want to get into the insights of C3PO, there's a lot of details inside this and there's the implementation of the integration pattern, the management of the context within the routes and so on. But the interesting part here is the concept of the components that allows you to do this integration with other systems and they've been able to transfer and pass the information along your flows. So this is the really interesting part that you really want to focus on. And on those components that are already available, as part of Apache Camel, to be able to simplify your integration. And at the beginning, we were talking about the Apache Camel umbrella project and the soup projects that are available. So if you already are familiar with Camel, you will have seen that they have been releasing now different runtimes for each one of the deployment options. So you can run in the traditional carafe mode using the Camel carafe project or you can deploy on top of Spring Boot. And if you check the previous session with Quarkus, there's another option to be able to deploy Camel on top of that container native and cloud native runtime. So you have also the Camel core project where all these internals of the framework are being developed. And if you're thinking about deploying Camel on a serverless environment running on top of Kubernetes, Camel K is the way to go. But in this moment, we're going to be talking more about the Camel Kafka connector sub-project where we are focusing on dealing with Camel and well as Kafka. So let's get a little deeper diver on the Camel Kafka connector project. And in this case, the idea that the Camel contributors have with this project was to tune their components to be able to play well together with Kafka. So the idea of having this part of the time where you need C3P to be able to easily talk and being able to mention and being able to speak what the dark side is saying. So you need to tune also C3P to be able to ingest and being able to mention what the dark side is saying. So this is very similar or what the Camel team did for making the batch of Camel connectors being able to play nicely with Kafka. And this is why they create this project. So the Camel Kafka connector project, it's a pool of Kafka connectors that are built on top of the Apache Camel components. The idea is to reuse in a very simple way all the different Camel components that are already available. The more than 350 connectors that are part of the project and being able to just add this logic and this tiny integration to make them run as if they were native connector plugins for Kafka Connect. And this was very successful because now we have connectors that are coming from the Camel components and most of them are already available. So there's a connector list that you can check with all the different connectors available. And they are being generated every time there's a new version of the Camel project. So you get all those connectors available and it's easy to use. And it's live as we mentioned a sub-project from the Apache Camels are really integrated. So if we think about why do we use the Camel Kafka connector? So most of the times I was saying you want to have your Kafka being able to talk to other systems either to get information from those systems or being able to push that information and being able to do a little bit more transformation. So remember there's two ways to do the integration with your system. You can use the traditional Camel approach where you need to deal with the company that retrieves information as its consumer or producer and you can do heavyweight transformation so you can use Disney approach to have a simple node code or local type of configuration using the internals of Kafka Connect. And it have three like use cases here. You have consolidated events that you are taking from a Mongo instance. And then you have several instances and you want to consolidate information. You can use the Mongo scene. So you can send information there for analytics or reporting. You can also use the elastic search sync that it's a component so you can use the elastic search rest API to send information from your Kafka clusters. And also for example, if you're getting logs that is a use case that people like for example in Telefonica they're having where they're taking information coming from their logs in a Kubernetes environment and they're using the connectors to be able to send information as an integration log and then being able to then produce a process and do a streaming analytic on those logs. And there's a ton of connectors that are very available, right? That allows you not only to connect to elastic search but also being able to connect to things like Slack. So no need to write Python code to connect to Slack or any other client. You can reuse the connector and just configure your tokens and then being able to send information into Slack or for example, getting data from Google Sheets or being able to send information there. And a little bit of detail on the code configuration. So remember, this is pure configuration. There's no code here. No need to write the Java DSL or the XML DSL. This is using the Kafka Connect approach where you just configure where is your class and the different parameters that your class needs to for the configuration. So in the first example, we have Amazon S3, then sending information using the configuration. You just need to pass the information on your secret and your access token and what is the region that you are being querying that S3 bucket. And if you're sending from Kafka to Amazon SQS, for example, is the taxing thing you need to tell the component where is the topic that you're listening to and then where is the path with the information on how you are gonna be accessing the SQS queues, right? And if you want to get started in your journey, there's a very simple steps you can follow to get started with this type of component. So the first thing is, you can go to the website on Kamal at Apache.org and they've been able to navigate into the list of supported connectors where you can find information on which of those connectors are available as the sync connectors or also as source connectors. So not all the different components are available as both. Sometimes some are sync, some are source, or sometimes they do as work as both. And then you can find the download links so you can download the components and then being able to create them as part of your Kafka Connect plugins folder or you can create your container images if you're running, for example, using StreamSync. And now that you have downloaded on sync the file, you can then take that set of jars, put in your plugins directory and then you can configure the connector. So in this example that you're seeing here, if you are using just traditional deployment of Kafka on a virtual machine or a server and you're using the standalone mode for Kafka Connect, you can just add the properties on your properties file and then start that process with that configuration. However, most of us will be using the container-based image, for example. And this is an option that is very common for where you're using StreamSync, for example, to deploy on Kubernetes or if you're just using the Docker compose configuration, you take a base image and then you can add the plugins either by creating your own image or just configuring your Docker compose. And then you can configure this new container and in the case of, for example, StreamSync, you can either use the REST endpoint so you can send that information directly as the endpoint or you can also define that as a custom resource because StreamSync uses custom resources to be able to define the different configuration of your Kafka ecosystem. So you can define a Kafka node or you can define a Kafka Connect cluster and also the Kafka Connect connectors. But if not, then you can use the REST API for that. And for that, let's see if we can do this in a few minutes and show you the power of the camel connectors. So please confirm me that you are able to see my screen and my demo. I can see only the presentation. Now it's good. Okay. Let's increase the font. Okay. Can you increase it a little bit more, please? Yeah, sure. Let's do a couple more. That should be better. Thank you. All right. So first I'm going to clone my repo and oh my God. So I prepared this example to show you how to do a simple trigger implementation of a timer that will send some information into our Kafka topic. So for this, I have this example where I have my camel Kafka connectors and here I do have a very simple configuration. First I have a Docker compose file that will start a local running version of a Kafka cluster. So if you want to take a look at the Docker compose, it's basically defining the suitkeeper, a broker using the examples from the code platform. It's using schema registry and I'm defining here a connect image that is using the Kafka connect data gen example from Confluent and the interesting thing here is that I'm adding here the plugins directory. So I will be mounting this as a file system in the same images. So when Kafka connected starts, it will query that information and then retrieve the information from the plugins and will automatically load my camel plugins from that place. So then the thing we need to do is go into the plugins directory and currently I don't have anything. So what we can do is then go into the page easily. So it's camel.apache.org and as I was mentioning you, you go to the documentation and you can find a different projects here, core, camelK, but we're interested in seeing the camelKaka connector. So we go to the documentation and you can check the connectors list here and we'll a little bit more documentation and how to get started. And here's all the different... Oh my God, this is a little bit odd. Well, something happened there, right? But yeah, it should be a table with the information. It looks like somehow the UI has been messed around. Let's see if we can find our connector. We're looking for the timer connector. Oh, never mind. I think I can download the pre-downloaded version because the timer it's... Okay, you know the demo gods are playing against me. So let's see, we'll have the camelK. I sent you a link on the chat if that helps. Oh, okay. Don't worry. I have an example here. So I'm going to just copy my into my plugins directory. So I have this and then I go to my project where it's got samples, connectors. So I'm just going to paste here my connector and as I was mentioning after you download the zip file that Octavio has shared with the chat, there's different jars that you can get from that jar and those are the implementation jars that you need to require from camel. They're slimmed down so you don't need to download everything and they will help us to put this. So now that we have downloaded the file and then we are able to start our environment. So we're going to do get back and then we can do docker-compose-compose-up for the recreate and this will start the docker-compose file that I showed you. It's already in use. So okay. It's okay. So my container is already running. It's right then. Now it's creating. As I mentioned, it's just running one node for Sukiber, one node for the broker, the schema registry and the connect cluster. So now that it's running, we can check that everything is there. So let's do a docker-ps and if everything runs correctly, there's the different containers running there. So now that they're here, let's do some integration. So as I set up at the Kafka Connect, we have available a REST API that allows us to interact with the Kafka Connect cluster and being able to query the information on the plugins and show the information there. So let's do a curl command to the local host, port 8083. That's the port that is exposed by the connect cluster. And let's check the connector plugins available. Let's make it a little bit clear using JQ. And here we have the default components available, the plugins, the connectors for Mirror Maker, for the file stream, the sync and the source that I was mentioning that are part of the default implementation. And now we have also the data generator connector that is the one that I've been using as a base image as well as the generic camel sync and the generic camel source connectors. But the one that we are really interested into is this connector, the camel timer source connector that is now available in my Kafka Connect cluster. So now that we've checked that the plugins directory was correctly read from my environment, then I can start to interact with the configuration. So for that, I have crafted a configuration file for configuring my timer. So we can see the timer.json file. We can see that we have a component that is going to be called timer. So it's the term that we'll be triggering every certain amount of time. I'm going to be using this configuration for my connector. I have the camel timer source connector. We'll be sending messages into a topic called camel.timer.one. We just have to put a name on our timer component. That's part of the camel configuration. So it's timer, and it will be triggering every 5 seconds. So 5,000 milliseconds. We're going to be using some more traditional Kafka configuration, like in this case, I'm going to use the string converter for the key, a JSON converter for the value, and some more configuration for the task of the Kafka Connect. In this case, I just need one trigger to one timer to be triggered. And these are the transformations that Kafka Connect provides for you to use when using the Kafka Connect framework. In this case, I'm just going to use the host field transformation to be able to create a JSON file that's called timer. So we have instead of just triggering with the null value, I just want to put some information like what is the topic that I'm using for the trigger, as well as painting or putting within the payload the timestamp of my trigger. So I'm going to use the host value, and then we're going to insert body transformation, and I'm going to be adding a field that's called TS for the timestamp and the information about the topic. So this is traditional Kafka Connect configuration. There's nothing related to camel here, DSL, and nothing like that. The only thing you need to know is these properties that are part of the documentation for the camel connectors. So now that I have go over this, I can then post to the REST API to create this connector. So let's run another curlCabat, and we're going to do a post method using the connectors endpoint, and I'm going to use this JSON file as the payload for my call. So if we run this, we have a successful result, and then it just says me that the timer was correctly configured, and it's now running on my cluster. So let's check if this is really happening. So for that, I'm going to be using a Kafka utility called Kafka Cat. It's a simple way to interact with your Kafka cluster without using the traditional console consumer, console producer scripts that are part of the project. So you can check this. It's pretty useful. And for this, I'm going to be reading from my local host, 1992, that's the bootstrap server URL that I'm using in my local environment, and I'm going to select the topic camel.timer.one, that is the topic that I configure to be triggering the information. So now let's take a look, and then you can see that it's going to be publishing the information on the timer and topic every certain amount of time. So every five seconds, we will get a new event, a new record coming into the topic that it's being triggered by my camel component. So sometimes you will need to trigger, you just want to get information every certain amount of time. You can craft a little bit on the data that you want to receive and so on. So it's pretty straightforward. This is a sync connector, but you can also use other connectors, as I was mentioning, to get information out of your cluster. So let's get back to the slides. Yeah, are you trying to interrupt you? But I have to warn you that you have three minutes left, okay? Correct. I'm already done. I'm just going to my takeaways. Good. So I hope you liked the demo. It was pretty interesting. And then for the takeaways that I wanted to share with you, it's that we have the combination of two really interesting projects from the Apache Foundation. For one side, we have all the power of the data stream platform that Kafka represents. And then we have all the knowledge and the accumulation of experience that the camel team has on the integration side. And playing them together allows you as a Kafka Connect user to get a lot of new integrations and a lot of options to connect to your system. And for the camel users, allows you to reuse all that knowledge that you have on the components and the connectors to be able to easily plug into the Kafka ecosystem using the very straightforward Kafka Connect approach. And finally, I want to show you some of the links that I use here. So you can go to the camel Kafka Connector GitHub projects. You can follow the conversation on the SELIF chat or you can also follow them on Apache camel for more in-depth information. So I hope you like this session. I think we are running out of time. I really appreciate the invitation to join you. And if you want to know a little bit more on what we are doing on the APIs and event-driven architecture world, you can follow me on Twitter, on at HGoretto with double o at the end. Or also I invite you to subscribe to my YouTube channel where you can find some more demos and examples and videos on how to work with APIs and messaging.