 Yeah, so hello everyone. Thank you for joining us to our talk bring your own bike over to the logging party. I'm Shander guba an engineering technical leader from Cisco formerly co-founder of bonsai cloud and I have like more than five years experience with Kubernetes and Fluent logging extra ecosystem. Please welcome my colleague at them as well. Hi. I'm Adam the dash I'm also coming on five years of experience With logging and I'm also with Cisco as a software engineer So for start, let's refresh our memories about Kubernetes logging. It's not too difficult because you can't configure anything The container on time writes the container standard output on the host file system and the kubelet service can read those files and the user can access them We are the cuba Kubernetes API and if you want to do anything more with your logs You have to deploy some subsystem to to handle these things and logging operator is one of I think the most interesting to to handle that under the hood. It deploys a fluent bit demon set which tails those files from your host attach Kubernetes metadata and Send all the logs to a fluent these tasteful set in fluent the we had a special plug-in called the router Which makes you able to route the different kind of logs based on the kubernetes metadata What it means is is if you have a label cause for example app engine X You will know that those logs will be engine X access logs and you can configure your log Pipelines accordingly so you can extract the matrix from them parse those files and at the end of the line You can send a wherever you want to do it and I could go on and on for hours about logging operator But it's it's not a deep dive on that matter I just want to highlight that it has a more than a thousand github stars that's driving community on slack with more than 500 members and if you are using rancher distribution of kubernetes under the hood you are using logging operator as well and Why I'm starting with this Out of west conference So logging is hard For example log formats are so different that you can't even like count them If you just check access logs like engine X Apache AJ proxy They have all the same information but in different formats not to mention about custom applications that you are Operating in different companies. They are like a whole lot of different formats and there are routers. There are appliances There are IOT so it's hard to make a common sense of logs and if you are go one step further You not just want to Pause those logs and post along the line. You want to extract value from it You want to maybe want to create metrics from an access log or you want to Unify different types of logs like HTTP or JRPC success rates and If you've prepared everything and everything works fine, there are still exceptions like gopani Java exceptions So there is another thing that you need to be prepared and If it's not enough you have always something special in your infrastructure There can be like special time formats that you need to unify or you want to inject a third-party Information into your log lines for example a typical example when you want to have Kubernetes metadata on the log lines you need to ask the Kubernetes API get the information attached to the log lines and It's becoming a common practice as well to use AI to process your logs So there are so many logs that it is really hard to have a common rules. What's good? What's not good for your system? So AI is becoming more and more important in this field as well and There can be any more so next year we may have like five different item on that list as well So just to summarize what is the requirements of such system? It needs to be performant We are doing work on the data plane and there is a lot of data when we are talking about logs It should be pluggable because if it's not you will have a big monolithic code base you have to manage a lot of third-party contributors you need to watch for clarity the compile times could go to the sky and with pluggable infrastructure you can simplify the complexity of the code by the making it a configuration in instead of code and Safety is also a really important part of logging remember logging usually handle sensitive data as well and and Why I'm talking about logging and now I will start to talk about when I think you know where I'm having so performance is crucial in West and the one of the main purpose purpose of Vesem is real life native like performance It's pluggable. I don't need to describe that we are talking about this all day safety, I think this is the maybe more important than performance itself and We have prior art how to use Vesem as plugins for example in envoy proxy You can use Vesem plugins to filter or authenticate HTTP traffic Vesem is language independence So you don't have to learn yet another language or configuration language whatever to achieve what you want and A plus one that it runs in the browser as well So we have theoretical architecture of subsystem where you have the log source on the left side And the logs are coming into the framework. It's a vast runtime And you have everything in Vesem plugins and transporting routing filtering all the way to the end So from a complex system we created a distributed complex system Which doesn't sound a really good thing to do unless you have the tools to operate and Observe what happens in your system and we will concentrate on that part We will introduce a tool that will receive log lines to WebSocket you will have some sampling and temporary storage to store those long lines and you can inject Vesem plugins into this system and Run through each lines that you want and check which plugins causes what output and what happens with them So Let's take an overview of these plugins how they work I think you see some similarities with the previous talk. We have a receive method on the plugin Which gets a length parameter it allocates some memory Which is specific to the runtime slash language how it does that It calls back to the host to get the input data at the address that the memory was allocated at the host copies the data there and The plugin then can process the input any way it sees fit and It can call set methods back to the host Passing the address and length of the data it wants to send to the next stage of the pipeline or as the results of the pipeline So demo time let's see this in action So our basic interface in the browser is this in the middle You have an input field where I can input anything Press run and you get the same thing. We don't have any pipeline yet. It's empty. So If I type something Run, it's the same thing now. I'll activate auto-run. So we don't have to run But press run all the time. So let's create a plug-in Here is a basic plug-in written in go. This is like the hello world of such plugins In the receive method. It's just checks if it got any data if it did then it allocates memory Calls get data from the host it receives the data and As you can see it's a reverse plug-in. It reverses the runes in the input data and If I compile this We're using tiny go to compile the code and I'll load it We get nothing There's an error in our code If I get back to the slides real quick I said that the plug-in is free to call send anytime it wants So as you can see the code we didn't call send so if we do and recompile reload Now we get our output that we wanted just the input reversed while we're on the topic of Calling send we can do this zero one or more times. So let's try that out Yeah, I just wanted to mention that zero time is not an error as well because sometimes you don't need a plug-in to pass forward Anything we might want just to increase a counter or do something with a third party. Yeah, it's basically an unoptimized drop plug-in So if I reload We now get as we expected the output two times If I load the same plug-in again Any guesses we get it four times, but it's reverse reversed. So we get the same thing four times now something a bit more realistic We also have an engine X parser plug-in Which is parsing engine X logs? So as you can see we have a few errors. This is because the code wants to have a Structured JSON message as input. So if I'll do that Now we get something. So as you can see the engine X parser is parsing the message field of the log and extracting key value pairs such as the agent the status code or the remote And such things. Yeah, so it's really nice to have This kind of tool set you have the text box You can try out each line of code you want but it would be much better to grab those logs from a production or production like system and Go through them because we want to simulate the the flows of a backend system in your browser and Back in Cuba Convalencia. I had a talk about the lock socket which is a tool to tap in the Production logs of a Kubernetes cluster. So it integrates with logging operator tap in the logs and fortunately push out on a web socket So now we have This architecture we have a Kubernetes cluster. We have looking operator installed We have log socket installed and we utilize a port forward to connect the cluster with this local machine And we can now connect and tap into the logs of a Kubernetes cluster on this cluster We run log generator which sole purpose is to Generate randomized agent access logs So get back to it. I'll just remove these reverse plugins because they are not doing any good So to connect to a web socket, you need a URL. Let's generate that URL And also I'd like to add this is not specific to Logs socket you can use any web socket URL here that is sending data will use that as log data So if I select any of these messages the input updates and the Auto-run runs our pipeline on the log line Yeah, so we have now production access logs, but what is the common property of access loss? They are boring especially the status cause they are just numbers But fortunately we had just the right plug-in to make it a bit more fun We brought the plug-in in assembly script, which will change the SS codes numbers into emojis So the 200s will be a smiling face the 300s will be a pointing finger the 400s will be a sad face And the 500s will be a sick face So let's try that out So as you can see we have an input and we have the plug-in the parser plug-in and after that the MWG5 plug-in and we have the status codes changed but one of the most Benefits of other assembly is to have more languages. So this was written in assembly script and We have another one written in go. I think yes and it will help us change the boring agent long strings into a more fun MWG5 agent strings and As you can see we change the status code with the agent text as well And we can click through the plug-ins. So check at which Plug-in what will be the output? So this is an important thing if you are building a long pipeline and you have an error in the middle Because you transferred something badly you can just click through and then get where the error happens But why the stop the madness here? We have another rust plug-in that will change anything that it's fine to emojis Yeah, so we have everything in place So we created the boring assess logs into good emoji for the assess logs And I know that this is not the most production ready thing to do and I don't advise to write it in production But you can get the idea where we are heading so you can do and inspect every logs in your production You can create your own well-established plugins You can try them out and you can build this whole pipeline in your browser And if you're done with it in your browser, you can push these plugins to whatever back-end system Then run on those trees and if you don't like working in the UI because you are CLI to maniac then you have an option to use our log test CLI tool as well and just grab those emojis from the assess logs So a few lessons we've learned while we were developing the tool and the plugins for it There are still few languages that have full wasm support There are many awesome languages that have some wasm support, but Few have the full support rust seemed to be the best option for us As we were trying to control the modules interface quite strictly Maybe cc plus plus has the same With golang We ran into some problems For example the built-in json package of go Uses a reflection by default, which is not really supported by tiny go at the moment So we couldn't use that we had to use a third-party plugin Assembly script, which is trying to mimic type script and java script It's missing json and regux support It needs also third-party libraries for this python ruby JavaScript and as i've heard C sharp and the clr has the most of the same thing it it has kind of an interpreter Which is compiled to wasm and this Really restricts the exports and imports Uh of the module Mainly to the interpreters functions. So it's not really easy to export or import your codes functions vasysupport is Hidden miss It's not available in the browser for obvious reasons And different runtimes implemented differently. It's changing Uh vasium models without generated js bindings Which is uh the default as we've experienced for most languages It's try the compiler tries to generate these bindings Untrivial to create and use if you don't want these And it's also hard to Add uh or no to remove those extra imports and exports that are generated by the compiler The bindings for wazzy and in golang's example The main function can be left out easily Yeah, so with this project our future plans Other than implementing the backend system for it as well Is like adding monitoring information like memory usage and execute Execution time to the plugins so you can measure properly what input gives you the best output The configuration so imagine if you want to build a production grade system from that you want to have vasum marketplace with plugins with different configuration So that should be included in this testing ui as well Creating stateful plugins is a bit harder. For example, if you think about batching the duping Handling multi-line logs you need to have some states in your vasum to process that And I think the lowest hanging fruit is stream processing in the browser It's totally the same that we introduce just automatically grab the latest logs and do the transformation on the pipeline In the last slide we have some useful links that have us to put together these presentation the logging operator Which is the backend to grabbing the logs the log socket which provides the website output From the logging operator flows You have the whole demo published on github so you can just Access it and try it at home everything that you saw here is open source And there is the link for the cube come well and see a presentation as well So you can get a bit more understanding about how logging operator works Yeah, so thank you for your attention Any questions? If not, I have some I love your talk. That was awesome I'll go So, you know logs are the raw material that drive operations intrusion detection security monitoring All sorts of downstream systems Is there any particular direction you're heading with with the technology? I would say that as we were working with a lot of staff and used a lot of tools to manage process logs The way how vans could fill in gaps with these plugins and performance system Is is a really good opportunity to make One go to for all I think it sounds too good But I have a feeling that in a couple of years that will be the reality enriching logs, you know Plugable metadata. I mean just over and over again Somebody that grew up in the security community and dealt with logs a large scale. I wish I had this 20 years ago That's for sure