 Good morning. Who of you comes from the embedded world? Please, almost a few. So I assume that the others come from data processing, right? Or you're not familiar that all the world stream processing. Ah, okay, the H side, perfect. Okay, we are going to get started. It's time, let me turn off this. My name is Eduardo Silva, and today we're going to discuss this presentation about IoT and stream processing. I understand that IoT is a very generic marketing term, but in general we can consider that anything that runs in embedded Linux devices or on some kind of DH. I'm a principal engineer at ARM. I'm the intern of this project, which is called Fluembed, which is an open source lock processor. And the goal of this presentation is to showcase the work that we are doing, mixing different components to solve data collection and data processing from an end-to-end perspective, meaning from IoT devices to the cloud or just do all the processing on the H side. And the main thing about IoT or embedded Linux sensors is that we got many data, right? Data is everywhere. Whatever that we have is generating data, could be your cell phone, sensors. Even if you have your spare Raspberry Pi running, you are generating data, CPU usage, metrics, memory, and so on. And depending on the market that we work on, it depends on the data that we care about. And actually the goal of has all this data, well data is useless if you cannot extract value, right? So if we have more tools or hardware generating data, we want to extract that value for monitoring, for business needs, for troubleshooting, or any kind of need. And in a simplified work, I'm using like a Raspberry Pi image, but just to simplify the hardware stuff, we get that they are generating a lot of data. It can come from sensors, there's a system service that is running on that specific box, but if you want to do analysis, basically you need to centralize all this information. Okay, nowadays there are just a few utilities that allows you to do all these analysis on the edge. Usually the traditional way is that you have any kind of sensors, get the base hardware, sending the data to the cloud, and the cloud you are aggregating all this data together in a database, and then you are doing your own analysis. And of course, this is a traditional way and sometimes works, but sometimes not, because usually you're assuming that you have a good connectivity in your exercise, right? And all of us could agree that network is not something that's reliable when we're talking about embedded devices. Sometimes things get down, they're not working properly, so we cannot trust that the whole data that we're generating are going to be stored in the cloud and we can do a proper analysis. Sometimes that is not the case. And there is a cute, quite few challenges here, because data come from sensors, can come from application and system services. Applications can be your normal application that is running in the device, a system service, if you're running in beta Linux, you can get data from system D, from the kernels, from, I don't know, from the proc file system. And all that information at some point you need to correlate it because you want to say all this data come from this sensor, from this kernel, maybe it's a bug or this application is not working properly. And in data ingestion, we consider that every hardware, every device that is generating data is going to send some kind of data event or look or record. It has many names for the same thing. We aggregate this and then we do the analysis. But of course we have some performance penalties. Even considering that if the network is working properly, right, the performance penalty is a performance in my feedback loop, meaning how much time do I have to wait in order to get some results or the result of the analysis that I want to perform. So this kind of latency and generates some problems because you would like to get some special alerts maybe in the order of milliseconds. But if you are trusting that your data needs to go to the cloud and the cloud is going to trigger an especially event to your mobile or to your own dashboard that will take some time. Assuming that the network is working. And in order to accomplish data analysis, one of the main problems is that we have a huge workflow. We started from collecting data, you had to filter the data, aggregate that data, then send that data to a database or a cloud service or whatever. Then we got the data indexing in the database space which takes time and then we got the data analysis. So if we have all these factors, there are many variables that can generate some problems. And one of them is data indexing. I'm not saying that data indexing is bad, but you need data indexing if you want to query your data. Because you have to consider that if you are from the embedded space, you're sending a bunch of records or data samples per second. But from the cloud space, they are not seeing just one stream of data. They are seeing hundreds or thousands. So in order to get all that information and store that information, it gets time. So you cannot get the results after you perform some data indexing. And this is where stream processing comes in. This is a concept on how you can query your information in a different way. And this is not about the age of the cloud. It's in general terms. Basically, stream processing is the ability to perform data processing while it's still in motion. Meaning the data is still in memory. If you think about a database, yeah, right, you get your data, your data hits the buffers, the buffers go to the file system, you have to wait until they kind of flash and synchronize that data to the file system, and then you can query your data. But here, we are providing a new way to query your data before to hit the database, before to store the data. And from a stream processing perspective, we need to have some special assumptions. Records, it's what is being emitted by an application, okay? But the composition of this is basically a timestamp, meaning when this message was generated, can be a sensor information, can be a service application message, and then you have, of course, the message. And of course, all of them come in a special structure. You can consider this as an example. This JSON, which has a timestamp, and unicycle some, and a temperature value, that is a structure message. And what the structure means, that I know before him, that the first, the key called TS is a timestamp, and the next key is temperature. But for a computer, this is basically just an array of bytes, it's no more than that. But we need to optimize also how we collect the data and process the data. This is mean keys and values. And if you have many sensors or many devices sending data, maybe the formats will be structured, right? But the keys and the values will be different, mostly the keys. You cannot assume that you will get all the same keys in one device and the other. So how do you optimize to query data that is coming from hundreds of services, devices with this kind of format? So the goal of stream processing is the ability to provide you a fast data processing without having a database with tables and no indexing, or which is called nowadays index-free, which is a new term that is kind of trend this year. Okay, how this works? We got the devices sending events, sending records. Here we're assuming that the sensors, you can assume that the hardware could be, it's collecting the sensor information or could be the sensor sending the events straight to the stream processor. And the stream processor is just a piece of software. And with the stream processor, you can query and generate real-time analysis results right away before to hit the database. This is not dependent, they can work in parallel. But if you have a stream processor, there's another complexity because you need to expose some kind of query language because you want to select some information for special keys or you want to filter out some information or you want to aggregate all of them. Because at some point, if you don't have aggregation in your stream processor, it could be kind of useless. And of course, you want to wrote this data because if you get all your data to the stream processor, you don't want to send out all the information back. You want to send just the results. That's what this routing is. And of course, do this all in memory. And this is quite challenging because this is done right now in the cloud space, but how do we solve this problem in the embedded side? This is like the traditional way. We have the DH, we have the devices, sending data to a cloud service. Could be a cloud, could be maybe a special gateway that you have in your environment. But this is how do we separate things. This is the traditional way. Another traditional way is that you have some kind of venerable software on the edge side collecting all the data from the sensors, from the devices, and then you ship this data out to the stream processor or to the database. This is traditional way, and I say that this is safest because this aggregator has persistency. We cannot say that our embedded device will have persistency. Maybe just collect data and send the samples of data out. But with an aggregator, on that side, we can take the data and ship the data out with a lot of reliability, meaning we have a storage, we have memory, we can do processing. And this is what people should do and that is fine. But how we can make it better because we are still seeing all the stream processing in the cloud. And if you're familiar with stream processing software, you can think about maybe Kafka, KSQL, you can think about Apache Spark, and all of them are Java stacks that needs to run in a big server because they are not lightweight. And it's something that you don't want to manage. You don't want to deploy. And maybe you want to do some kind of simplest processing and you are dependent of a big software that you don't want to manage. And this is where login comes in. We're talking about the embedded stream processor and login and all of them are very connected. And this is the final goal of this presentation. For example, if we talk about login and login, I'm not talking just about our syslog or syslog. Logging in general, I aim to solve the data collection problem. Pretty much the same thing that we have in embedded because data can come from TCP over the network over the UDP file system and from sensors too. If you run any kind of Linux box, you will see that any kind of service generates its own login format in own events. But they have a different format and likely they don't have a structure. If you look at the Apache logs, it's quite familiar. You understand that you have a timestamp, the HTTP method and all the other information. But if you pass that information to the computer, it's just an array of bytes and that's it. But if we want to do some data analysis over that, we need to process that information. We need to filter that information. There is some extra work that needs to be done. A simple message like A, open source summit and A generates a common message with metadata because that message also was generated at some time. You need to have a timestamp and also it was generated from some stream. On this case, this is an example from a Docker container. It's not related to embedded but it's pretty much the same thing. If you send a message from an embedded device, a firewall, using syslog protocol, you will get a timestamp, you will get maybe the hostname and the message. And appending metadata is more complex. If you think about how this works in the cloud, in the cloud you have more metadata. If you think about a Kubernetes cluster where you're running with containers, that special message becomes something like this because you need to have the pod name, the pod ID because it's not about the message, it's about the context of the message that is being generated. So logging in general is complex. Data processing is complex, it's not that easy and of course it's expensive because you are parsing text messages and of course they are binary protocols but anyway you need to do processing. And to solve all these problems is usually we implement some kind of data pipeline where you have an input where you collect data from, you parse the data because maybe you have to convert from instructor to structure messages then you want to filter the data to a pet metadata or maybe drop some information. You buffer your data and then you send your data out and the outputs can be a database, a cloud service, a file in the file system or anything that can be used to store the information after the pipeline. And this is the introduction and this is FluentBet, one of the project that I maintain and work on it because it's time to solve all these login challenges. It was created in 2015 originally for Embedded Linux but quickly it evolved it as a cloud solution for clusters, for Kubernetes. It's fully open-sored, it's part of CNCF of the FluentD project and it's really in C-language, we have a low-memory CPU footprint. It can run some 500 kilobytes but well, disclaimer, 500 kilobytes doing nothing, right? Because if you have data ingestion, you're doing data processing, of course you're going to consume memory, of course you need CPU. You can connect different plugins to collect data from different input sources, parts data or different output plugins to dispatch the results or the output to different backends and of course supports TLS and network IU. In login, this is the same thing as we took it at the beginning. We got an application that generate an event or a record. This record has a timestamp and a message but internally we handle all of this in a binary format with message pack. Message pack is like a binary JSON so it allows you to serialize the information and jump between keys and values easily. You're not going byte by byte usually as you're doing with JSON and it's quite performant. And when you get the data in, this is I'm talking a little bit about internals, you get the data grouped by tags because maybe you are listening data from a file system from Apache but also you are getting data from your syslog endpoint. So you can listen from data from many input sources. Then we aggregate that information internally either in memory or in the file system and then we roll the information out to a database. This is like a login mechanism with the pipeline that I just explained it. One of the adoption is like 200,000 deployments every day in the cloud, which is a lot. You can consider that fluent bits is deployed more than a million times a week. So we can not mess up things. And it's run by AWS, Google platform, Datadoc is going to start using it too. So and the good thing about this kind of tooling is that despite we are solving a problem for the cloud, we see that we have the same problems in the IoT and embedded space. But of course we have some requirements, right? Because you're not going to run this kind of tooling in a sensor or in a really constrained device because it needs an embedded Linux. And what we're proposing here is kind of login on steroids, something that it was working and we have a proof that works in the cloud but now can work also in constrained environments. And we care about performance and for the embedded side performance will be how fast I can get the results from a data analysis that I want to perform. So actually this is what the traditional way, you access the file system or your retrieve data, you process the data and then you wait for data indexing. And the thing is we want to avoid that. And this is the introduction to the new stream processing capabilities that aims to solve the problem that we just discussed at the beginning. We used to have this kind of model before on the edge. We have this log processor connecting to an external stream processor and doing all these things. And now we have all these stream processor capabilities internally that can run on the edge side. You can think that this is kind of a, it could be in the gateway in your architecture. But also it can be improved where the same stream processor, the same service can run on the same device. And what would be the difference here? If you think of this model, the hardware or the device is sending all the data samples to the stream processor to the service, to the log engine. And that's fine. But if you go with this model and your device has more capacity, you are just sending the data that you care about. And when I say the data you care about is about calculate the results of the data samples that are being generated internally and then send out the average number of temperatures or, I don't know, the number of error messages or something that you're really interested in. And the stream processor is just implemented after the storage engine in the pipeline. So if you get the stream processor, at that point you are able also to ingest the results as an input source of data in the pipeline. That can be routed out to a database or a cloud service. So how do we implement this? Basically, we aim to implement all of this using SQL queries, but there's no database, there's no tables or anything. Everything runs on memory and everything is written in C language. So we can do this kind of data stream selection. I'm going to do a demo right now. Key selection, you can select the stream. You can run aggregation functions over a period of time. Or also you can say create a window of five seconds and calculate the average temperature of a specific device and just send the result of that calculation outside and not every sample of temperature that we have. And also we are able to create streams. If you are familiar with Apache Kafka or KSQL, you will see that this is a pretty familiar statement. Actually, we are taking the best ideas that are implemented in the cloud, but trying to bring all these features to the edge, to the edge side. For the edge, I'm talking about embedded Linux or that specific area. In the first demo, we're going to have a normal file that has one million records. And these one million records, I'm going to just query all of them that belongs to the country, United States and numbers equals mayor.80. Maybe you can say, hey, this is pretty simple. I can ingest all the data into my database. But in order to do that, you need to perform, create your own program that. Collect the data, transfer the data and ship the data out. We're going to use a query like this. I'm going to show how Fluentbit works. Simply, we're going to create a stream called Results with attack results because we're going to group the records and select this IP country and number from a stream which is called open-source summit and with the conditions that we have at bottom. Okay, this is a special configuration of Fluentbit. If you focus here, we define an input section. That input section is defined that it's going to consume data using the tail plugin. Tail allows you to follow a log file in the file system and going to put an alias for this data called OSS Summit. The path for the file is data.log and we're going to parse this data because the regional format of this data is JSON. But JSON is not binary, it's just text. So if we look at this, I'm going to query data.log and use jq for formatting. Going to break it. This is kind of sample of the records that we have in the file. This could be anything. The data comes from your sensor, from system D, journal D, or anything like that. And we are going to query this information and I'm going to load one streams file where I have my SQL query. That file streams.com is the same one that I have just here in the slide. We have the created streams, we select and we have the conditions. Okay, here we're running the results that we are getting from the file, locally. This same thing is something that you can do in your embedded Linux service when you're collecting, I don't know, a lot of sensory information or system information. But here I'm sending the results just to the standard output interface. What is most interesting if you can send that data out to a database? Another demo that we have is that anything that is most interesting is how to aggregate information. Here in the terminal, I'm just sending all the data that it's matching the condition that it had in the word statement where they belong to country United States and numbers is greater than 80. In this second demo, I'm going to run this query which becomes a bit longer. We're going to create the same string called results. I'm going to select country but calculate the average of num. So we're going to summarize and calculate the average. But every five seconds, that means but every five seconds for the whole data that is flowing through the pipeline and calculating the average and by grouping all the results by country. Usually what you do to accomplish this is that you ingest all the records in a database and run this query. But here we're skipping all that database, all that indexing, all that stuff. And for that, I have a second strings file. You can have many streams with queries and all that stuff. This is a very basic demo. This is the same thing that I just got in the slide. But what I'm going to do, I'm not going to use this demo now. Instead of sending the data to the standard output, I'm going to send it to a remote fluent bit. You can think that it's a database or anything like that. So on the right, I'm going to run another fluent bit instance that we're listening using the forward plugin which listen for messages over the network. And for every message, I'm going to send it to the standard output. I'm going to format the data in with JSON lines every one second. Okay, so on the right, we have like a kind of server listening for messages remotely. And we're going to do here on the left is to use my output which has the same name forward because we have output forward input forward which is our binary protocol for communication and send the results there. We don't need a format. And then we're going to change our streams file. Okay, that should be fine. So every five seconds or so, we should get the results on the right side for the average calculation of the numbers. And that's where I send the data. So it's calculating one million records but every five seconds is aggregating that information and sending the data. Here are the results. I'm going to create a window and here are the results. And as you can see, we just send two records to the remote endpoint and not one million. That's the goal of this stream processing capabilities. It's not about a different way to do logging. It's not about replacing your RCSlog or your mechanism but it's a way to implement another service that can connect to any kind of source of information in your device that can be anything and then do data calculation and just send the data as the results out of the box. Okay, this is aggregation. A new future that just we got merged two days ago is the ability to perform some kind of basic machine learning stuff on the edge. Okay, and for that, you can use simple linear regression algorithm to forecast, for example, any kind of data. On this example, we're going to forecast the memory usage of my computer here. So based on an algorithm and using a query based on my own time series data that I'm consuming, I'm going to decide how this will operate. And if we're going to use a different configuration file that I call forecast, which is different because here in the input side, I'm using the input plugin, which is called Mem for memory statistics. I'm going to run that plugin without stream processing so you can see what kind of information generate. So every one second, I'm going to collect memory metrics from my system and send it out to the standard output. So we get the memory total, the memory use, the memory free, the swap, and so on. So we get snapshots of memory every second. I'm doing this locally. Okay, so if you want a more computing format, JSON lines, and we can do something like this so you can get a formatted data. This is what we are getting locally. Okay, but what we want to achieve is to forecast how much memory I'm going to use in the future. And there are quite algorithms that you can run. Right now, we are only supporting forecasting right now, but we are planning to extend that. We're going to see. So I'm going to collect the data from the memory input side. I'm going to send the data to an output, to the standard output, but also I'm going to send everything that match forecast to the remote endpoint. Let me show you the query here. And creating a new stream which is called forecast. That the results of that will be tagged as forecast. So when you tag the data, you can set the data that it has attacked to different endpoints. So I'm going to send one of them to the standard output for all the JSON lines and the other to the remote endpoint. And this is a query that I'm using. Selects, I'm invoking the function, time series forecast, a timestamp, memory use, a number, from stream memory and every 10 seconds. So every 10 seconds, 10 seconds, I'm going to forecast how much memory I might use. And why this is useful? Because, for example, if you're running, you're listening for data from sensors and your sensors are measuring temperature, and you want to say, A, I want to not monitor what's my temperature right now, but I want to get an alert once my temperature could be greater than 50 degrees, but I want to get an alert beforehand. Not when that happens, but when it likely will happen. And for that, you need these kind of forecasting functions. Okay? So everything that matches memstat will be sending to the standard output, which is this. And everything that matches forecast, it will be forwarder to the other flowing bit that's running on my right. I'm going to restart this to clean it up. Okay, and here, oh, let me show you the, this is a query for forecasting. Yeah, we need two queries here. I'm not going to explain the better, the first one, but the second one is the same one that we got in the slide. Here we go. So we are getting a snapshot of memory on the left. And every 10 seconds, we're going to get the result of a forecasting of memory on the right, which would happen now. So, of course, this is sending the number of bytes, right? It's not doing the full math or formatting that value. We're still implementing more functions. But the point of this is that you can implement anything to forecast any kind of time series, generate alerts, and of course, it's up to you how you implement alerts, right? We just take the data, we do the processing, we send the results, and it's up to you to decide, hey, every time that I get an alert, please trigger an HTTP call to a service or send me a message to my phone or anything like that, or a Slack, I don't know. So you want to know, right now I'm forecasting every 10 seconds, but in a real case, you will put a minute or 30 minutes. Yeah, because the forecasting is calculated in the window for time, okay? When this happens, and I see that, okay, this value is reaching a certain limit, I'm going to get the results. And based on the results, you will take some action. I can explain further after the talk? No, no, with linear regression, you cannot do that because you're using your past time series of data. I understand that it's a new kind of linear regression improvement that allows you to do that kind of things, but that algorithm is not implemented yet. And actually, if you are working with stream processing or know how these things works, will be ideal to get those proposals because we are not experts on stream processing, but most of these ideas and results and implementation come from people from data management that has a strong background on that, and you as people from the embedded world say, hey, I need to forecast this kind of information or group the information in a different way. Okay, that was the presentation, and if you have some questions, please raise your hand. I think we have a mic here, I'm not working. Do you have any questions about stream processing, the edge? We support window tumbling and hopping windows. So you can do also the forecasting with hopping windows, so if you see that you don't get the results every 10 seconds, you increment that period of window. Not sliding windows yet. Tumbling and hopping windows. Yeah. Yeah. No, no, that is something that is in the roadmap right now. Yeah, we just missing the sliding window part. Yeah. Yeah. Yeah, actually, there's one specific use case that is mostly in the cloud. For example, if there are certain, for example, databases, they charge you per amount of data ingested, like Splunk, for example. And if you take the use case of Splunk, they provide you login tools to ship the data from everywhere to the database, but they don't allow you to filter data out. Yeah, and that's why you pay a lot of money. And of course with this, you save a lot of money in the payables in the cloud, you optimize your network bandwidth because you are not sending thousands of data samples, you don't saturate the gateways, the switches, and anything like that. So a good point, yeah. Yes, we don't provide a framework to write plugins, but FluentVid has all the plugins that are built statically linked, but you can write your dynamic plugins. For example, let me show you a bit of the source code. If you go to the plugins directory, we have many. Okay, in means input, filters, filters. For example, there's one interesting filter which is called Lua. So if you don't want to write your filter to modify special keys and values in C, you can write it in Lua, and load that as a simple configuration file. And that is a premium set, very use case. Also, if you want to write, you don't want to write your output plugins in C, you can write it in Golang because we have a Golang connector. So you just write your Golang output plugin, build it as a share library, we have documentation for that, and you just link it at runtime. We are getting a request to have this in the input and the filters, but switching from C to Golang is expensive too. So we are trying to see if people move to C or have a different approach. Yeah, the question is, this is mostly for the edge or the cloud, how did it compare with others? And you have to compare, for example, a stream processing Apache Spark Kafka. They need a JDM, a Java virtual machine. That is quite heavy, it's quite expensive. And you need at least one gigabyte of memory. Here you can run it in two megabytes, right? Because we are caring about data processing and not the whole storage capabilities. That is one thing. From a licensed perspective, there's no issues as Apache licensed, same as Kafka. It's not about a replacement, but an extension on how to approach a problem in a different way and get your builds down. And when I say the edge, the edge can be anything, right? Here I'm talking about a gateways or embedded Linux devices or a Kubernetes node that can be the edge too. Yeah, there's not like a straight way to do that. For example, if you want to discard data that has certain patterns, you have filters for that. Because in the filters, you can say, hey, drop this record. You got it, I don't know, I don't need it. Maybe it's a debug message. You don't want to send debug messages to production database. But also you can do it with SQL queries. You can say that all the data that you are sending from the input side is not rootable. Meaning it's not going to hit the output destination. But all of them are going to hit the stream processor. And the stream processor can say, hey, create a new stream of data for all the data that match a certain criteria. So you can play with the filters and the stream processor. We have, okay, the question is about if we have examples to use the Fluent Bit API. One thing that I didn't mention about Fluent Bit is that you can write your plugins or you can use Fluent Bit as a library because it builds as a library. So on top of that, maybe your C++ program or if you write, I don't know, Python binding, you can connect to the library and say, load this plugin, this filter, this thing and start in a separate thread. And it will start your own login pipeline. Let me show you the GitHub repository. If you want to write your plugins in Go, we have this Fluent Bit Go is in the documentation. And here we have examples on how to write, for example, an output plugin in Go. And it's documented how each callbacks works and how it operates. If you want to write it in C, we have this plugin. So this repository Fluent Bit plugin. Here's one example. This is a simple example and let me, okay. We basically, when you define a plugin, you have callbacks for initialization, what to do upon flush time and an exit callback. The STD out plugin is pretty simple because it just get the data and print the data out. Nothing else. There's none that were I owe, there's nothing. If I think that you want to learn a bit more of the internals, so if you want to stay for this part, that's fine, if you want to leave, you can leave. This is like an extra bonus that you always has just in case. A bit of the internals Fluent Bit so you can understand how it operates. Basically it's a single thread, even loop that get data asynchronous. Everything is synchronous internally. There are no blocking calls, neither with the file system, neither with a network I.O. So one way that we implemented in order to have a reliable connectivity with output plugins, you can say if you're going to most of the time when you send data out, you go through the same process, like create a TCP connection, convert the internal data to the desired format, write the data over the network, wait for a response and so on. But things could fade at any time. But the lines in red are the blocking calls. And how do we work around that? This is an example of the elastic search plugin. And what we did is that in our own API, for example, when we're going to flush data, we're going to create a connection, we use coroutines internally. So we have a full asynchronous I.O. mechanism with that main evan loop and all of that is abstracted with our own API. So on line eight, when you're going to issue a connection to a remote endpoint, that function say for you, this is like a function. It could not stop, but internally it's stopping when it's connecting to the remote endpoint and it stayed waiting for the kernel to notify when the connection succeeded or got an error. The same thing on line 21, when you're trying to send the whole HTTP request to the server. And if each plugin can say, also have special return values and return value means I was able to flush data, I need a retry because if you were trying to send the data and the network was down, you need to retry later. So we have a full retry logic because you don't want to lose data. And you can have errors too. We have plugin helpers, so for timers, for HTTP client, or anything that you need in order to connect to a remote endpoint. The plugins that we have to listen to get messages from the input sites, kernel messages, serial interface, tail for file, system D, in total there are more than 50. And you can write your own, this is an example of a filter reading in Lua. It's like six lines in a configuration file. And also if you care about metrics and you want to monitor how the data pipeline is working internally, how many data I'm getting, how many records in gestion, you can connect to its own HTTP endpoint and gather the metrics. This is using Prometheus, so you launch Prometheus and you let Prometheus pull metrics from fluent bit when it's running. Yeah, that was the content. And if you have more questions, we can talk outside and thanks for coming.