 Good morning. Good afternoon or good evening for whoever's joining this conference. My name is Ricardo Ferriero, and I am about to start the presentation for the open source summit, right? I would like to thank the open source summit conference for inviting me for this for the session and Looking forward to talk with you and during the Q&A. All right. For those of you that probably apply for the session We're gonna start talking about some very cool interesting subject called event screen processing and more importantly We're gonna talk all of this in the context of using Apache Kafka Which is by far one of the popular few is for engine for how to implement event stream processing these days, right? So what I'm gonna do now is start sharing my screen because mostly my presentation is going to be a live demo I'm gonna explain a little bit about what I'm gonna do for the next 45 minutes, but Ultimately what we're gonna do is to build a event stream processing application for a scratch so I'm gonna start sharing my screen and I Would like to All right, so you should be able to see my screen now and I Hope looking forward to so before we start it Let me explain a little bit about myself so for those of you that don't know me my name is Ricardo Fuera and I Work as a developer advocate in this company that we call confluent, right for Conflict is basically a startup from California But is mostly known because it is one of the companies behind Apache Kafka as the open source community Right as well as it is a company that builds a event streaming platform on top of Kafka So that's that's how we have this different relationship with Kafka, right? so as a developer advocated by his job is to primarily connect developers with knowledge right developers with content, so if you ever have any questions about Kafka specifically or the event stream platform that Conflict develops, I would like to help you right to accomplish that and My job is also being part of the Kafka summit program committee members So Kafka summit is an annual conference for about Kafka and it's all about the open source version of Kafka, right? So in there I helped to build out the final agenda with the other program committee members, right? So these are my contacts This is my email from Conflict and this is my personal email as well, which also happens to be my blog so I usually publish some very interesting context that right there about Kafka or event streaming and things like that, right? And this is my Twitter handle so for those of you that use Twitter Please follow me on Twitter as well as these two other handles as well Which is the Kafka handle and the Conflict handle with usually there are some very good content that are published on Twitter And I think is interesting for you to start following those those handle. So as I mentioned before the whole purpose of this presentation is to give you an Idea about what is an event streaming platform or an event streaming application looks like right? so I am a developer probably just like yourself, right and There are some concepts out there that are not easy digestible, right? And I strongly believe that event stream processing is one of them, right? I've been working with this technology for the last three years, right? And even today I still have a little bit of problem to fully understand the technology, right? And as developers, I think the best way for learning a technology is to look firstly as an example Right now, so show me an example of an event stream platform so I can actually realize What I can do with it, right? And then if the internet is is Applied we can actually go deeper in the Understanding the technology and learning all the bits and bytes of the technology and we can become experts on it, right? So that's the whole objective of this presentation and the way we are going to do this I have built this demo that I've called streaming Pac-Man, right? So Pac-Man is a very famous game I don't need to actually explain what the game is all about, right? But what we're gonna do actually is I have deployed this game in a cloud provider, right? And I'm gonna expose This game for you for everybody that is listening to this session, right? And you were going to have the opportunity to play with the game, right? So what we're gonna do it what I am going to do actually is to Allow you to play and as you play events from this game is going to be transmitted to a Kafka cluster That is also running on the same cloud provider, right? So why is that? Because I'm gonna grab all those events and as they happen I'm going to process them in near real time and come up with something called a scoreboard, right? Just like you you've seen here in diagram, right? So it's scoreboard the interesting part about it is going to be dynamic What that means is that Whoever is actually having a higher score is going to be put in the first or previous position Regarding the other players, right? And this is going to change automatically, right? So instead of building a static set of data such as a table, right? Although I agree that it looks like a table, right? But it's going to be a dynamic table, right? So that's the whole beauty of event stream processing You can actually start acting upon events as they happen And start using start doing something useful with this data, right? You don't need to rely on the I have a problem saying the term owed, right? Because sometimes Some people might be still working with this philosophy of how to handle data, right? But in this context of event stream processing, we can refer to the old days of First starting the data into a database so you can actually be able to process them, right? I know that there's a lot of applications that still do this, right? And there is nothing wrong in doing this type of architecture, right? But there are specific use cases where you don't have the possibility, right? To wait until data is being stored until you start processing them There's a lot of use cases that are drawing specifically on the context of cloud, serverless and event stream processing that needs A different architecture principles to how do you go going to handle the data? And this is what we are going to accomplish with this streaming backman demo, right? So Let me make sure we're ready to go We are and everything that I'm going to show here is actually an open source Implementation that I have done and I made available on github So this is the link where you can grab all the source code That I'm going to use it right now in this presentation, right? And the architecture behind this What the source code provides, right? It's a end-to-end event streaming application that is composed by two main pieces And this is the first thing, for those of you that never worked with event streaming before, right? Event streaming is highly composed of two main components, right? There is the one component that we call streaming data or data streaming as you want to call it, right? And there's another layer called streaming analytics, okay? What is the difference between the two of them? Streaming data is all about data acquisition Ingestion and storage, all right? So in other words, you're not necessarily processing anything yet You're basically acquiring data and storing data, right? And this is what the architecture does Because we're going to expose the application and the cloud provider just like it as you can see here And when you start playing with this application Events from the application will be invoking this API gateway What everything happens to be running on the same cloud provider So for the sake of where everything is running and communicating to each other It's within the same cloud provider, right? So the application is going to submit events to this API gateway The API gateway implementation backend is going to be a lambda function, right? And this lambda function basically receives the event and write into a Kafka topic, right? So basically that's all about, just like I said, event data ingestion and storage, right? So once the data is there on the Kafka topic We're going to use an engine that happens to be another open source project Along with Kafka that we call KSQLDB, right? KSQLDB is largely known as being a stream processor, right? That uses the SQL language as a way to express your string processing applications But underneath what it does is actually deploy a bunch of Kafka streams Applications that actually pull the data and it's going to process the data As they flow through Kafka, right? So that's basically the high level definition of KSQLDB It's more than that, of course But for the matter of this demo What we're going to do is using KSQLDB To implement this second later that I've mentioned before called streaming analytics Right? So what is this streaming analytics? Streaming analytics is the ability to act upon data that is still in motion, right? And you are producing value about that data, right? So what we are going to do specifically is to All the events that are coming to the inbound Kafka topic We're going to use KSQLDB to process them in real-time You know that real-time actually doesn't exist or apply And we're going to produce the output of that processing That streaming analytics processing into a second Kafka topic That we're going to call it scoreboard Because the name is very suggestive, right? And this scoreboard will be made available For a lot of different web channels Primarily a web application, right? That's going to be running on the cloud provider But also through APIs, you can actually access the scoreboard Rather calling the API directly or using some other devices That I'm going to show in the end And that's the prestige part of this presentation This is something that I'm keeping myself a secret I haven't put down on the abstract When I posted this session for the conference But because I like to do a surprise By the end of this presentation About how some smart and intelligent ways Can you consume the data from the scoreboard, right? So I'm looking forward to provide some fun as well Because as I said, you're going to be able to play with the game But I need you to understand what we are doing with this application So as I said, we're doing two separate sets of processing Streaming data, right? Or data streaming, as I mentioned before And streaming analytics, right? So in other words, we're going to do ingestion, Store, and processing of data in real time So I would like to invite everybody That is listening to this session To grab your devices wherever you're wearing right now Like your mobile devices Or maybe you have a tablet, right? You can use them to scan this barcode over here, right? This is where I'm going to provide the game itself, right? You don't need to actually stall any application This is basically a web application Running on a cloud provider Technically speaking, what I'm doing is creating a bucket On that cloud provider And the application is basically a bunch of HTML JavaScript and CSS files That I've uploaded to that bucket, right? So could you please scan the barcode And you will be able to do this And start playing the game And for those of you that would like to use computers for this What I'm actually going to do just to help you out Is to grab the direct URL of this application, right? And I'm going to make available here On the chat for everybody, right? So if you're looking to the chat Actually, let me pass again because, okay So you can if you're using your laptop For watching this conference, right? You can simply grab this URL That I just passed on the chat And you can put on the browser And there you have it You're going to be able to play with the game So let me show you how that works So you can actually see how this happens So I'm going to literally use my browser And I'm going to pass it here The URL that I've just provided in the chat So as you can see Once you've accessed the URL You're going to be presented to this initial screen Of the Pac-Man game, right? And you're going to enter your name It doesn't have to be your real name You can be a nickname Or something that uniquely identify yourself, right? Just be mindful about not using some jokes That might be offensive for Because we're going to show this All everything that you're going to write it here Is going to be shown on the scoreboard, right? So please be mindful about And respect it to other people, right? So I'm going to provide my name Specifically here, right? And click play So once you do this You're going to see this quick animation That basically shows the Pac-Man logo, right? And all the ghosts for the game Are going to have specific nicknames For the Ponchi Kafka So if you have a Ponchi Kafka background You're going to recognize some of them Like the broker, partitions, controller, and the offset, right? And if you click again on this You will be actually starting your own game, right? So I'm not going to play here I'm going to press the letter P for pause, right? And because, of course, I'm here to present another play, right? I want you to play So I'm going to go back here for the slides So for those of you that are using mobile devices Or want to use mobile devices Just scan this barcode over here And you're going to end up with the same URL That I've just provided in the chat, right? So I'm going to give you one minute here For those of you who get ready to play the game And you can start playing, right? And then after that, I will go back to the screen And I will start building the scoreboard from scratch, right? And of course, explaining step by step So feel free to scan the barcode Use the URL and start playing Yeah, so Edward asked it for about what is the link So it is in the chat But I'm going to provide here as an answer as well For the Q&A So everybody can follow up on that So thank you, Edward, for your question Okay, I do hope that everybody at least Grabbed the URL or scan the barcode So what I'm going to do now, very quickly Is to observe if events are Heating that coffee topic that I've shown before, right? So that will be the proof that whoever is playing the game Right now, the events are being ingested And stored into Kafka, right? So the way I'm going to show this I'm going to grab this URL over here What happens to be the endpoint where you can connect Technically create a client session To the ksqldb cluster that is running on the cloud provider So that's what I'm going to do right now I'm going to as a client I'm going to connect with that cluster And once connected, which It seems that I am I will be able to query those streams So let me just see if there's any questions That Q&A isn't showing up for all the right Okay Yeah, so Laura is saying that the Q&A answer is not showing up for all Yeah, Laura, so I think you could actually just go to the chat On the platform and the URL that I've provided is there So because the Q&A will probably only be available In the end of the presentation, right? So you would be better off to just grab the URL from the from the chat, right? So let me check if the events are coming So I'm going to print the topic user game from the beginning And here we go It seems that we're having people having lots of fun playing the game right now So this is what we call event stream If you have if you never kind of a fully understood the concept of event streams before This is a perfect way to understand because An event stream is basically an unbounded set of data, right? It's a set of data that never ends if you if you will So the way There are different architecture styles that you can use to develop our software these days Right? One of them right is the opposite of event streaming Which we what we call data streaming or static streaming Where the data is first acquired and Permanently stored into a database Whether if it's a relational databases or no-SQL databases And then you're going to come up with programs that you're going to write and different programming languages That is going to first step number one read the data from the store and step number two Load the data up and then start processing right Events streaming applications are different, right? Events streaming applications need to deal with something like this That never ends, right? So how do you process data that is continuously being updated? And it keep coming right every second every millisecond every microsecond And you're still going to be able to process all of them And instead of using batch Variated styles, right? You're going to use some renewal real-time style of processing data So this this is the exciting part of events streaming applications, right? And you're going to see that's really easy If you at least know SQL or as well You're going to be able to express events stream processing applications really easy using ksql db, right? So i'm going to stop this. I know that everybody is playing I'm going to stop basically what i'm doing here with this printing here Probably the game will still be available and if you if you fancy to keep playing the game Just playing the game. I don't mind, right? What i'm going to start doing now is building the scoreboard, right as I promised, right? So um This is stream of events that I've just shown it here It belongs to a topic called user topic. Sorry user game. Okay. This is a topic called user game That's what I did here when I when I Print user game. I'm basically this is a command in ksql db to dump Whatever is contained and still coming into that topic, right? This is called print, right? But this is not cool because when you print stuff or if you dump stuff you basically are getting access to the raw data, right? What we want to do is to transform the data into a format That we can start Dealing with it, right? We can start querying it. We can start filtering it We can start transforming that data to whatever you want, right? And the first step to build an event streaming application is to come up with the stream representation of your event streams, right and Coming up with the actual representation that you can handle with the platform. So What i'm going to do now is to reset the Offset for those of you that know Kafka Offset is the jargon for you're going to go back to the beginning of the top So whatever you can do now is going to reuse all the data that has been processing since the beginning of the times, right? and What i'm going to do is to create the two streams one of them is called user game And the second one is going to be user losses, okay? We're going to create them and i don't know i'm going to explain which One of them does right user game holds your Game data right such as your score your current level or the number of lives that you have right? And the user losses is every time you game over into the game This is going to represent a loss, right? Which is not good. I know but This is going to represent this is going to be another event That's going to be emitted to the topic user losses And this topic is uniquely used for counting the number of times you lose, right? Why this information is important? Because later on you're going to see that the score board will be based on this information, right? For example, imagine two players have the same score. How do we pronounce which one is the winner? So, okay, if they have the same score, we can actually use how many of them actually, um Didn't die so much, right? So that's a way for for For to figure that out, okay, so With that said, I've just created streams and look at what I'm what I'm what I am able to do with the stream I can actually select them as if they were Uh database table, right if you will But here's the difference between a database table and a stream in case equal db as you can see here. I'm selecting Uh a stream of events that never ends, right? So but now instead of accessing the raw format of the application I'm actually accessing some data that has some data types Has the notion of columns and each column has associated with value of it, right? So If you go back to the presentation, you're going to see that this is what I've done here Right. So this is the raw definition of the user game, right? So I have the field called user in another field called game And this field game actually has a inner JSON payload on it So the game has a inner JSON payload that in turn has score lines and level So what I've done here is to create a stream called user game And then I'm mapping this field here to this user field which happens to be a bar chart, right? And the game field I am actually designing here as a struct or a data structure if you will, right? So a struct is basically something that can have Interviews whether you can have older structs on it or just like a here I have a flat definition of the game itself And then once you do this you can actually Query specific set of beta directly like this for example, if I if I only want to see the user And the score I'm going to say game and then score front User game can make changes, right? So doing this I'm going to be only selecting the user column and the score column Is everybody playing? Right now probably not because it's not showing anything, but anyway, it's it's it's fine, right? The point is you can actually see You can actually access specific fields from the game field, right using this notation here Which is a pointer if you will for those of you that came from C++ It reminds the pointer when you navigate through the the hierarchy of the the object, right? So that's how you come up with this notion of selecting specific pieces of data, right? Um, all right. Let's move it on so We did the same thing with the uh user losses as you can see here where's my browser Or my code Yeah, so we did the same thing with the user losses topic, right? So we We brought the representation of the user losses as a stream So what I'm going to do now is to out of those streams I'm going to create two tables, right the first one is going to be called stats per user And the second one is going to be called losses per user, right? So stats per user. What's going to do? I'm doing some aggregation here. So in a per user basis I'm going to calculate the max score and the max level of that particular user and I'm going to group by that user, right? Losses for users are going to be the same So I'm going to actually uh in a per user basis count the number of times that they've they've lost in the game So let me select this right here and I'm going to Connect again to case equal to b and I'm going to pass this statement over here So I can come up with the table stats per user and the table losses per user Okay, they have been created Okay, good good good good So let's see if they have been created indeed. So if I show the streams I should be able to see the user game in the user losses streams and if I show the tables I should be able to see that Uses per user and stats per user table. So what I'm going to do now is to play with a little bit I'm going to select everything from stats per user stats per user We can see the type of data that actually is there, right? I'm sure if everybody's playing it with the game right now, we should be able to see some data coming in But if not, that's okay Let me just stop the square over here One of the interesting things about queries in adventure and applications that as you can see here once they start They never end you have to explicitly kind of a Contro C or stop the query so they can start So they can stop pulling the data from the sources, which in this case is the stats per user table, right? sticking a Y here for We stop the query That's way a couple minutes so we can stop. Okay Okay Let me grab the url of the case equal to b again and instead of querying from here this we know what I'm going to do is square from here I think we're going to be better off Using it's our own terminal to handle that specifically because I can increase the font size, which is great, right? All right, so let me print Let me show which topics we have right now Um, sorry show topics It's a let me print stats per user From the game It seems that it's sticking a wire to fat the data from the cloud provider I don't know if that is do some high traffic for data coming in or If maybe I have I'm having internet connection, which I doubt it, but In any case, it's always good to see if we're not we're still connected, right? As we are so it's not about internet connection. Maybe it's traffic. So Let me do this We're gonna cancer this for some reason my key security be cluster is not responding. So Uh, I think that's one of the cool things about doing live demos, right? We can actually see Problems coming up and how The speaker is going to solve them because ultimately That is what you would have to do, right? And that's the beauty of it. So I'm gonna actually, um Come up with these, um, okay that's the terrible Time to actually need to update the password, but let's try it It's like my gateways time and out for some reason. So this is not good All right, let me see what's happening with the uh with my cloud provider Okay, so Let me log into aws. What happens to be my cloud provider right now and I need to provide my code No, no, no, no, let's do it. That's the coolest part All right, so let's see What's happening with my, um key security be cluster It should be up. Yeah, that's where I saw it. So something is actually My key security be service How many tasks is we have I have two running which is good. Okay So, okay Connect again to the service Since I don't want to have I don't want to lose too many time doing this. Let me create the final scoreboard table So we can actually Start doing what we came here to do which is build up the scoreboard. So The scoreboard table is basically it's going to perform a join between losses per user and stats per user to come up with a third Output table and that's the table we're going to use for accumulate all the results. So Let me create this Hopefully it will create if it creates it means that the key security be closer is Effectively running, um, which is good it is So i'm going to create this final table over here as well We calculate the highest score possible Which is among everybody that's playing is going to basically create calculate the max score for it So now that is working Let me check this right. So let me do this with this windows over here It's not about this windows confusion, but um, I need to see both the game As well as the scoreboard that I've created alive. So what I'm going to do here is essentially Access the scoreboard. I have an hml page Better It is working My queries from my computer are not hitting the cluster But I can clearly see here that for for everybody that is actually playing the game right now The scoreboard is somehow Being calculated. So we have currently 21 people And oh, I'm in position 19 right now with my terrible score in the play game So those are the people that I actually right now playing the game, right? And what you are seeing here is the result of that last table that we've created the scoreboard table Remember this one that does basically this what the scoreboard table does It comes up with the data from stats per user And joins with the losses per user, right? So let me try to select one more time. Uh, if it doesn't work, it doesn't work. I'm not gonna insist on that but Because I can see that it it is working what I Possibly cannot be doing right now is to issue queries To the cloud provider from my computer. Don't ask me why I really don't know But uh, let me try to select the scoreboard from here scoreboard Need to image change for that. Sorry image change Okay I don't know it is working now So I can only assume that the cloud provider had a hiccup on the load balancer because The way I expose kisik would it be to the outside world if through a load balancer, right? So maybe the old bars got overloaded. I don't know. Sorry about that. But What mattered is that these scoreboard Table, right is being automatically computed or should I say recomputed continuously? Basit on the stream processors that are behind Each stream and table that I've created so far, right? So in order for you to fully understand how that works and that's the part that I really want to touch base Right if you ask me the whole purpose of the presentation is to come up and what I'm about to say right now, right? which is If I show my tables over here, right, you're gonna see that I have the Like I said the scoreboard table loss per user stats per user in the highest score, right? So if you describe this table extended Scoreboard for example, which is our main table over here I can't describe show queries then Okay, so probably I'm not being able to show the course. Okay, here you go So let me just minimize a little bit because it's too different size. So you see that I've had this query Over here called CTA scoreboard 41. The name it doesn't matter, right? But basically, this is the query that is as a background service Continuously running, right and it's fetching data from the sources, which is stats per user and losses per user and Flushing the output of that processing into a scoreboard topics, right? So one of the things that we can do is to it's to say in case it could be explaining this query, right? And then you are going to be able to see that what this query does is to basically Create this execution plan over here where all the data from these Sets per user and losses per user are being mangled, right into a unified a table that happens to be scoreboard, right? and Basically, this is the processing topology that behind scenes Everything is a processor in the world of adventuring processing, right? And behind scenes, this is the topology that has been created out of those statements over here that This is the beauty of ksqtb and other string processing engines as well. You can actually express your string processing behavior Using this high-level programming language, it happens to look like SQL although it's not SQL, okay? But anyway, the complexities about how to implement this topology has been abstracted away from you And that's the beauty of ksqtb, right? Okay, so let me go back to the Time here. So we we have some time. So that's this is good so Let me check here real quick If there are any questions, I can see you there blah blah blah, okay, the game the game works, right? It's just the scoreboard which doesn't show it anyway. I share the game Okay, good. Good. This is working so far. We're gonna have some time for q&a in the end. So don't worry uh What I would like to do now is to basically Show you some interesting ways to make that data available outside, right? So This is second Okay, all right, so Where's the scoreboard? Where's the scoreboard? Oh, yeah, here we go. So the scoreboard is over here So as you can see here The scoreboard actually looks alive, right? Um, and as I said, this is a simple hml Code that I've just uploaded on the On the cloud provider But what this page is actually doing is pulling a web api, right? And I'm going to actually show you This web api so you can see What is what I did? So this is aws, which is the cloud provider that I'm using it And I've deployed on aws and an api using the their api gateway service, right? I call it the scoreboard api And if we look to the stage of this api, we're going to be able to retrieve the endpoint of this api So this endpoint over here is the same endpoint that this web application here is to every five seconds Pulling right to retrieve the data from the scoreboard. So what I'm going to do now real quick Is to call this api so you can actually see that the data that is being shown on the application Matches the output of the api results, right? And this is and this is this api in turn is reading from the scoreboard topic Right, so I'm going to create a new request. This is going to be a post Right slash scoreboard Scoreboard and if I hit send Here we go. So I'm going to put the two windows here in the Same All right, so as you can see here. So the first user winning the game is fox neagle, right? And the second is rogie join you I don't know what that means and then the third one is going to be daniel lozaro, right and fourth So I keep her and you got it right So as you can see here, it is the same data and that that is a reason for this because this api Is actually reading the same source of truth Which happens to be kafka, right? So When you design event streaming applications kafka is used for both data ingestion, right and the data Output right so the output in here is the scoreboard topic Okay, and who Populates this scoreboard topic with the data is the stream Processor that has been basit on key sql dp, right? So it's very simple. Yeah, if you came from the database and backward It is the same the same job you would do in a database, right? You would come up with a program that reads from the search table You would do something with that data and the output of that Something it will be flush it into a output table, right and that output table the downstream application Would start reading from it, right? This is the same principle, right? This haven't changed actually, right the only difference is here Is that you don't process all of this in a batch mode You process them in as they happen in your real time, right? So Before I actually start the q&a session because I know that we're Almost getting there, right? Let me show one last thing and I do hope That you are still able to see my Screen over here my face and my hands moving here Actually, can someone give me the feedback if I you are able to see me Or not, okay, I will Then hope That you can see me right because what i'm about to show you Needs for you to see my live feed my face and everything, right? because Okay, let me pull this up You're coming up. Oh, thank you. Thank you. All right, so Not sure if you are seeing here, but what i'm showing here is this device This intelligent device that i'm not called by name because otherwise i'm going to activate it by voice, right? But you probably know what device is this, right? And so what i'm going to do is to ask this device for details That are basit on the scoreboard that's been computed by case equal db, right? So I'm going to actually put this aside for a second So we can actually see the scoreboard and hopefully you are still being able to see the device So look what i'm about to do right now, right? So a lax Tell me who is the bass player The best player is fox and eagle All right, let me turn up the volume a little bit because not sure if it was too loud. Uh, let me ask it again, alexa Tell me who is the best player The best player is fox and eagle It seems to be the same data, right? So let's try something different alexa Tell me who are the top five players The top five are fox and eagle rojoin toy keeper daniel locero and louis vierra So the point being proven here is that I have made available the same set of data through an api and also a device that can consume the same data, right? So that is to show you that once you start applying the answering process and application You can make the data available elsewhere, right to the wider application Intercept on that data, right? So that this could be easily some application that you would Implement for coming up with some smart ways to deliver data, right? This is just of course a very Sealy example, but the point that i'm trying to prove here is that When you start leveraging event streaming applications as a concept, right? amazing things can start happening so What i've gone here Let me just go back to the slide real quick so we can start the q&a what i'm done here is this i made data available, right? So this is a scoreboard cable and then what the api is doing is basically Searching the data into a cache that i've used at radis in this case because i truly like open source stuff Right and then i have this engine that also happens to be part of kcco db We call kafka connect, right? So It uses a connector basal approach to continuously pushing the data that is existing on the kafka, right to something Right, it can be it depends of the that's something depends on what the connector does I use a connector for radis, but it could be a connector for elastic search It could be a connector for kcendra. It could be a connector for dynamo db, right? Whatever you want to make it the data available. There is a connector for right And what i've done later on is to use it the same principle But the the device right for an amazon is actually Using this accessing the same radis cache that has the data from scoreboard topic Right, so with that said let me pause for a second And let me jump into the Okay, let me jump into the q&a. Uh, we actually are for the sake of this demonstration It we're done So whoever has a question right now, and I do hope that you have questions, please You cannot just ask them So there's one you're um from Paul I was able to get 5 000, but my name is not there as this was easy. There might be immersive images in the scoreboard Are you playing the old data? That's a good question Okay Yeah, we can see you we can see you this is basically the feedback if you're See me alive. Okay, good. Good. Good. Good. Good. Good. All right. Um Is there anyone have any questions that I could answer? Okay, so there's one here from Michael Erickson So it's kind of the best streaming framework to use for collection data from sensors um I Michael I would say yes and no, okay And why is that? Kafka for the sake of scalability purposes and being able to handle large volumes of data It is probably the by far the best technology for it, right? However The way you are going to actually collect the different sensors And bring into Kafka that's the part where as a developer, right? This is a developer feedback You you might be better off using some technology some ads such as mqtt, right? um So you could you usually what I've seen the field, right in my experience is that you have a first layer of mqtt brokers That are your sensors actually are proving to the mqtt broker And then this mqtt broker kind of a relay all the information to a kafka closer that where you can do your stream processing right, uh, so you might be asking all right. What's the problem of? Publishing directly to kafka. So the answer is The kafka client api although is very simple to use it it requires you to have a durable tcp stateful connection, right? and usually sensor doesn't have this ability to be always on right, so It is possible technically speaking right, but I would recommend to use this first mqtt layer, right? So for addressing that All right, so okay, so Joshua asks it can just be used to update a state on it clients game Joshua, let me think um Yes It would be more like implementation Design than actually something that is possible not so the answer is yes, right? How I would do this it would be I could use Another table that holds specific states and my my client could actually keep pooling locally That particular state its own state from the game, right? So wherever the string processing is trying to do specifically for one session It would update that session and then my client would go there to grab all the results that has been Processed for it, right underneath and behind scenes. So yeah, it's possible, right? The point here, I think that what is most important is Data is is constantly moving around Kafka topics, right? So you can actually come up with another set of string processors that will Okay, scarboard is doing what is the best which is holding the data for an aggregated matrix Right, but I could use the data for specific users to do something else The whole point is you would do this flush into a new Kafka topic and your Specific session would go there and query right and by the way, and I'm glad Joshua that you asked this question It is possible on ksqlDB to this Right, so you can write code that can query data from specific tables And you can do this directly from your application and in a matter of fact, this is what I'm doing here Uh, for example, you see here that the highest score is fox and ego and has Dude, this guy's good 254,000 right right now and It's coming up So in my application, let me show real quick this here There is the javascript code that I've used to calculate the highest score Here So look what I'm doing here. I'm basically coming up with a query, right? This is javascript. So this is no big deal I'm coming up with a query to this table called highest score that I've created for by last And I'm performing a http post over here to that query as you can see here Okay, and I can grab the results from it. So whoever you process them your application can query directly using ht post and literally four days ago I the ksqlDB Um Project they coming up with a java client. So you now are able to if you're a java developer, of course You can write code to query directly ksqlDB and do Amazing stuff right Hopefully I answer your question Joshua Uh, okay. Thanks, Michael. Thank you too. And then Why ask it is the source code developed for the sample streaming app developed? Yes. Yes Let me put in chat here It is on the presentation which the conference will be make available later on But if you if you want to right now, I can grab to you No worries. It will be my pleasure. I'm going to pass here on the chat, right? Actually, I'm going to answer your question. So Okay, just pass it here and save And I'll go back All right. So the source code is there. Thank you And then there's another question. How can app code query from ksqlDB tables? Um Wait, I don't not sure if you saw what I just show it But I just show it how so I'm going to go back here in the code So there's this section of the code where I need I need to come up with this I need to come up with this behavior over here. Let me show you So I'm going to log in with my user. I'm not going to play. I just want to emphasize something, right? So can you see here that Although I have my own score over here, which is one hundred ninety very low, right? The highest score from all the user is being shown here, right? Who calculates this continuously Is that high score cable that I've created earlier? Using the ksqlDB code, right? And then what I'm what this browser is doing is to continuously pulling Right performing this query over here Against that highest score Right. So that's basically you being able to issue ksqlDB queries directly to ksqlDB to retrieve pieces of data Right. So yeah, that's this is possible, right? Hopefully that answer your questions. Uh, go ahead Okay Is there any other questions that I could happily answer? Right Do you like the application? I hope everybody enjoy it. I think the whole purpose of this Streaming Pac-Man demo is because not only is fun to play Right, but also gives you the foundation to understand the the moving pieces, right? All right. What what is this data streaming? What is streaming analytics, right? How ksqlDB kind of a glues together or all the pieces of data that that's the interesting part, right? So let me double check again if the Um, okay, apparently not. So I think we'll Probably end this presentation in a matter of seconds Thank you for everybody that participated Thank you again open source summit conference for inviting me into the session and I'm looking forward to Seeing you out there And be safe, please