 Hello, everyone, and welcome to Cloud Native Live, where we dive into the code behind Cloud Native. I am your host, Itai Shakuri. I am the director of open-source at Aqua Security. I'm also a CNCF Cloud Native ambassador and the host of today's show. So every week, we bring a new set of presenters to show you how to work with Cloud Native technologies. They will build things and break things and answer your questions. This week, we have the InfloxDB team to talk with us about Telegraph, Kubernetes operator. We have quite a crew today, so they will introduce themselves in a minute. I just want to remind everyone first that this is an official live stream of the CNCF and a such a subject to the CNCF Code of Conduct. So please don't add anything to the chat or questions that will be in violation of that Code of Conduct. Really just be nice to each other and the presenters as well. So with that, I'll hand it over to Pat to kick off the round of introductions. Cool. So we'll go around and do a quick round of introductions. So first, I'll start with me. I'm Pat Gon, engineering manager at InfloxData of the deployments team. So we're the team responsible for continuously deploying our product into Kubernetes in the cloud. And with that, Foycek, you want to introduce yourself? Okay. Hi, my name is Foycek Kocian. I work at InfloxData. I work in Pat's team in the deployment team. And I'm among other things. I'm one of the maintainers of Telegraph operator that allows deploying Telegraph services in Kubernetes, which I believe Pat will explain in more detail. A little bit, a little bit. But Foycek, I actually didn't say it. So I'm located in Portland, Oregon. Where are you, Foycek? So I'm living in Krakow in Poland and I'm working remotely. Cool. All right. Jess, do you want to introduce yourself? Sure. I'm Jess Ingrasolino and I'm the engineering manager for the Telegraph team at Inflox. Cool. And then last but not least, Popey, chess grandmaster. You want to introduce yourself? Hello. I'm Matt. I beat chess, Patrick chess once and she's salty about it. I'm Alan, also known as Popey. I'm a developer advocate on the Telegraph team and I'm based in the rainy United Kingdom. Cool. So that's the introductions. So I'll just keep going then. Is that OK? Yeah, please go ahead. Cool. So Foycek is going to actually work on the Telegraph operator. He's going to share his screen and he's going to do that. And before he does that, I'm going to I'm here to quickly introduce a couple of things that will be relevant to his live coding session. So what Foycek is going to be working on today is the open source Telegraph operator, which is hosted at github.com slash influx data slash Telegraph operator. I'll leave it to him to describe what changes he's got planned today. But I wanted to quickly explain what Telegraph is and why the heck we want to Kubernetes operator for it. We've got a couple of members of our Telegraph team, which we already introduced here to answer questions, but I'll keep it light for now and leave room for questions or conversation during our session today. So please do ask questions. So Telegraph is influx data's open source plug-in driven server agent that you can use to collect metrics from a wide array of inputs and write them to a wide array of outputs. I think you get the idea. There's a lot of choices here. So I checked earlier and there are close to 250 plug-ins now. So Jess, your team has been busy and your community has been busy. So Telegraph can collect metrics from a lot of different sources, but for today's discussion, the focus is going to be the input is Kubernetes. On the output side, there are also a lot of different options, but internally influx, I don't want to shock y'all, but we tend to use the influx DB output plug-in. So that's I think what we're going to show today. I do want to mention you can, there's a lot of different output plug-ins that are available if you don't want to use influx DB. The Telegraph operator packages the operational aspects for deploying a Telegraph agent on Kubernetes. It's deployed as a sidecar container based on annotations and provides a Telegraph configuration to scrape the exposed metrics all defined declaratively. It allows you to find common output destinations for all your metrics. So in one place, you can say where you want all of your Kubernetes metrics to go on all your application metrics. So that's I think really quite useful. The outputs are called classes. It generates Telegraph configuration by combining all the different pieces, the configuration outputs assigned as classes along with the pod annotations that describe what data to scrape and how. And yes, I did read that because I wanted to prepare for y'all ahead of time. So I'm hoping that when Vojcek is waiting for something to build or start, we can press him on some of the details of what I've just described. And with that, I hand it over to you, Vojcek, to do as it said, to build and break things. So go forth and build and break things, Vojcek. Thanks a lot for the introduction, Pat. Yes, so what I'm hoping to be doing today, assuming nothing breaks, is to extend the Telegraph operator that we talked about to allow hot reloading of configuration changes. So there's been a recent PR and I'm just going to probe it out here for a few seconds that allows detecting and hot reloading of Telegraph itself that I believe is either released as of today or will be released soon. But that has been quite a recent change, which allows Telegraph itself to detect the changes. What we want to do now is for those of us that use the Telegraph operator to inject the Telegraph into Kubernetes deployments so that we could take it to the next step that if you change a configure class, as Pat mentioned, in a Telegraph operator configuration this change gets propagated all the way to all of your Telegraph site card containers. So I'll start by explaining, and that's in the contributing documentation, so I'll show that. I'll start by explaining how we develop Telegraph operator and what I'm going to be using today. So obviously we have a lot of tests and these are really important, but the most important thing is being able to run the code we're going to run live in a Kubernetes cluster. So for that we have a pretty much ready to use setup of using Kubernetes in Docker, so kind. And I just happen to have Kubernetes, a kind cluster deployed already. And I happen to have a build of Telegraph with that PR that I just showed, which is important because again, it may not have been released yet and I didn't decide to trade the last minute. So I came prepared with my own tribute image. So right now what I'm going to do is I'll explain what the change will be. So as I mentioned how Telegraph and Telegraph operator connected. As I mentioned, we have the logic to detect configuration changes. The way Telegraph works is it has a config file and the way Telegraph operator uses it is we have a secret, a Kubernetes secret that has the final configuration generated and then that's mounted as a volume for Telegraph. And Telegraph operator works in a similar way that we have a secret with the classes definition and that is also mounted. So in order to do this and to end, my goal would be I want to detect when the configuration of the classes changes and write code that's going to find all the secrets managed by Telegraph operator, check if they need to be updated and if they do, go ahead and update them. So that is the plan. I'll start by creating the two different parts of the change that I think I will need. So I think I'm going to need something like a watcher which is going to be watching the file system. The reason is because Telegraph operator is reading the files as the mounted volume. So the natural way of interacting with it would be to just use the file system and what we're going to be using and I'll just import it for now just so it's there and I can have it in here already, is we're going to use a package called FSNotify which is a cross platform package to get notifications for file system changes. So that's going to be the first part that I want and I'm going to make a comment of what we'll be building. So it's always visible. So logic that monitors Telegraph classes directory because it's a directory that just mounts the secret and calls a function when it changes. So this is what will be in this file and we will also create another file which I will call updates.go and this, sorry about that. And this file will have logic to find all secrets managed by Telegraph operator and update them. So that is the first thing that I want to do but while, but I also want to show how what I need to do to have the changes because that will allow me to explain the development workflow of Telegraph operator which I think is quite interesting. So we have a make target. We use make files and that was also mentioned in the contributing section, my time field, yes. So there is a make target that just builds and maybe I will simply show this kind build that builds the image and then we use a feature of Kubernetes in Docker called side loading that will load the images as we've locally into Docker into the kind cluster. So when I'm going to deploy the Telegraph operator I am going to be providing my local image. There's also one thing that I have prepared. I prepared, so we, the repository provides a ready to deploy YAML file that allows developing that allows running Telegraph operator for development purposes. So it has a static self-signed set of certificates which are needed because Telegraph operator is using mutating web hooks and I will get to that in a bit. But we have that ready to use. I simply modified this so it passes some settings to the Telegraph operator specifically using my image that my kind cluster already has side loaded and to tell Telegraph to monitor the file system for changes because that's not enabled by default. At least as of now since the latest Telegraph doesn't support it yet. So now that I've built it, I can just apply my deploy, deploy, watch. So I can deploy my Telegraph operator and I can see that it's running already. So the way Telegraph operator works and I'm going to deploy an example of Redis so we can get started with how it works. I have my simple Redis deployment. It has some annotations specifically how to get the input and this is a snippet of Telegraph configuration and it mentions the class which is the type of application that this is. I'm going to create a namespace test which is what just gets deployed and I'm going to deploy my Redis example. So what we'll see right now is while this YAML file has only one hot listed, only one container listed, we see that it has two containers because Telegraph operator has added the Telegraph sidecar container. So this is how Telegraph works today but now what I'm really interested in is doing the reloading logic. So in order to get bootstrapped with this, we're going to be cheating ourselves a bit initially. So I'm going to create a structure for the watcher which we'll be using and now I can actually do this which will be using a watcher. Just notify the watcher against this. Okay, we will also be keeping the directory and we will be, I think that may be all that we need for now and we can, oh, we also want to have the callback which you can say which you can define as a type. So this will be, this will be just an empty function and I'll explain why in a bit once we get to the implementation but basically this will just be called whenever we see there is a change happening and as I mentioned, we're going to be, we're going to be, oh, we also want, we also want is a logger. That's what would be nice if we have that. For some reason, Mike, okay. So we want to get the logger and we also, okay, this is what we really want. Okay, so what we want to get is the logger and the telegraph class directory and the unchange and obviously this would return my structure for an error. Oh, we fold this for a bit so we can see more of the code. So we're going to, let's create this. So create this into copy the logger into, well, we don't need that directory but let's keep it in here. We can clean it up later and unchange. Okay, so for now, there's not much logic in here. That should be fine except they don't know why my, oh, okay, I need to go get, go get com, okay, maybe let me just reload the window. Wait, Jack, what are you using FSNotify for? So FSNotify is a package for detecting changes in the file system. Okay. So as I mentioned, we, since we mount, okay, I will also show the classes. So we have the classes which are the, which are the outputs that we talked about. So we have the classes and this is a Kubernetes secret. And then in the deployment of our telegraph operator, this is mounted as a path in the telegraph operator. So because this is mounted as a file system, we want to watch whenever this changes. So one of the things, one of the nice things about mounting secrets as volumes instead of using environment variables is that whenever the secret changes, it takes a few seconds of up to a minute from experience, but the change gets propagated to the pod. So we want to monitor this directory to see the changes, but we're going to start a bit smaller. I really don't know. Okay. Now I know what it was going to be like. And let's go, okay. So we were doing this, but instead of actually watching the file system for now, as I mentioned, we're going to be cheating a bit. So we're going to write a, of course this watcher, going to write a dummy function, which is just going to be sleeping and calling the callback every 10 seconds. So the reason why we're doing it this way is because now we don't have to implement this. The whole logic for the watcher to implement the other part, which is the updates. So we can focus on the updates, which I believe is more interesting. And then we can fine tune the monitoring. Okay, so. Vojicic, I see we have a couple of newcomers that have joined possibly after we did the introduction. So could you just very quickly recap the context for everyone who joined recently, what it is we're doing and yeah. Okay, yes, definitely. So what we are working on Telegraph operator to add functionality to support hot reloading. So Telegraph operator injects Telegraph sidecars into pods that have specific annotations. And Telegraph is an agent that monitors different types of metrics and scrapes data from applications and can now put them into multiple supported types of outputs. So what we want to do is we want to support the case that if I go and change my settings, which we call and change what we call classes. So one of the types of applications that could be monitoring. If I change something in here, I want that update just updating the secret will end up in Telegraph operator and Telegraph getting to the point where they transparently get the latest configurations that I just applied and reload the configuration. So Telegraph starts using the new settings instead of the old settings. And the rationale is that maybe, and I don't know if you mentioned it previously, the rationale is that if you're monitoring a lot of pods and systems, and we do, we use Telegraph operator in anger. So we have, one of the issues we had is if we ever change settings like we want to change a token or where the data is sent or like you want to send the data to multiple databases, we can have a mirror of the data. Before the feature that we're going to be implementing, we would have to go in and force a restart of all of the, at least all of the Telegraph containers. And often this meant restarting all of the pods. And what we want to get to is that all of these configuration changes happen transparently. And as an operator of a class, I could just go edit the settings with Telegraph operator and it gets propagated to all of the pods that need to change applied. I know we're pretty excited about using this internally because this is something like, because we use this operator in our product and we'll make a change. And we're like, okay, now we're waiting for it to be seen. So the- Yes, this is actually a feature that came up from our internal issues that we're having. So this is an improvement that we think the community would benefit from, but this is something we would definitely benefit from. Okay, so now that I've created my thing, my changes to watcher, I can initialize this. So new Telegraph classes watcher. Let me create new name watcher. So I'm going to create a new instance of logger. I'm going to pass the Telegraph classes directory, which is a flag. So a command line option to Telegraph operator. Yes, in here. So it defaults to exactly how we mount the volume, but if someone wants to use Telegraph operator differently, it can change. So we want to pass that. And then we want to pass our callback, which will be nil for now because we need to implement our callback as well. So now let's just add some basic error handling. And now we don't need the watcher. So setting thing up, watcher failed. Okay, and colon is right, no new variables. So that's great. So now we need to do something similar for the updates. So secrets, let's call it updater actually, which will mean that I need to also take the Docker file, but somehow I don't like name updates. Okay, secrets, let's call it secrets updater. And what will we need for that? So we'll definitely need client set because we're going to be talking to Kubernetes. So let's just Kubernetes clients. Okay, that was, that went well. I was, we need logger as well. And what we will need is an object that's, this is an object that, this is the part of existing part of Telegraph operator that's handling generating, that's handling adding the secrets, managing the secrets. So we need that to call some of that logic to update the secrets. So we have our structure and we also need a function that can create it. So we're going to also take the logger, the sidecar. And I don't think we need anything else also. This should be a point to the server handler and we will return the secrets updater or a point to the secrets updater or an error. Okay, so now we need to initialize the Kubernetes configuration because Telegraph operator is running within the cluster. We could just initialize this by default. So let's just config, get config. Oh, okay. And I'm happy to see that VS codes with all of that. And it's on, I didn't have to think too much about the names of packages. Last time I tried it, that didn't work as well. At least with generating the configs. But okay, so we have the config. Now let's create the client set from config, config. We have all of that. And again, if there was an error, just return the error. And now I believe we can return the secrets updater. So we want to get the client set. We want the logger and the sidecar. And obviously new. Okay. Okay. And what we will create here is the onChange function. So we get the, and that function will do nothing to now. It's just, well, it could just say update secrets, checking for updates. And that could technically be all and could even deploy it at this point and see if it works. We also need to create it in main. So like data and error. It's new secrets updater, we take the log with name. We take the sidecar, which is already initialized above. So we can just use it. And that's it. So if there was an error, that's updates failed. And now we can pass the callback that we've just created. And I think at this point, we can even try to rebuild it and kind of build and get it deployed. Make sure the things that we see that log, just also show how to deploy a new version of telegraph operator. And I can actually start to implement the changes while that's happening. So in order to find all the, so what we want to do right now is find all namespaces and find all secret telegraph operator managed secrets in each namespace. So that's what we're going to be doing. Telegraph operator already adds labels that indicate the class name and the pod that this secret was created from. So we could use that to find all of the secrets. And that's why we created the client set. So basically whenever we see changes, we're just going to find all the secrets that telegraph operator manages, regenerate their configuration. And if they change, just go ahead and update the secret. So let's just, just, okay. Let's just, before I continue with that, let's update it. So even though the make kind build, so I run the make kind build which builds the image and side loaded it into kind. But in order to update the deployment of telegraph operator, I have to go in and first delete it. So let me deploy my, let me deploy telegraph operator again. And hopefully somewhere in my history, I have this. And I can keep my logs of telegraph operator below and we should see every 10 seconds the new message we added to the secrets we updated. Exactly. So now we can continue with this call to client set. And I believe that was, client set, that was call view one, call view one namespaces, okay. Not get that list. Also, we want to create a context. And let's add it to do this timeout to the context. So this would be useful that we could deal with timeouts in the future. But okay, for now we just want to list it, give it the context. And I believe we want to do this. And there is one thing that I always, that Visual Studio always does, Visual Studio always does for me and I keep on renaming it. I always prefer to be just to be called meta view one because then the next inputs for view one gets a bit more complex. So we don't have the watcher handle error. So in this case, we will do something like this. If error isn't new, we'll just log it and exit. We'll log error, give it the error and then we say unable to list namespaces and return. We don't have to be more verbose because the logger already has a different name. So if the namespaces were listed properly, we can iterate over all items. And now what we could do is we could call a function that will implement update secrets in namespace, namespace name. And, well, okay, updates secrets in namespace. Okay, let's do this. Let's use namespace. So the way the way the logger works is that you can pass a key value list after the message. So I can say this is my message and then I can say this is the namespace and it's going to be included in the logs. So this way, whenever any single update of any single namespace fails, it will not prevent other namespaces from failing. So this is a better way of doing this and also then our function for a specific namespace can return errors, which is useful because the logic in there may be a bit more complex. So, secrets of fater. So let's write this function. It should take the same context that we have. It should take a context because we're going to need it. We call the client set and then let's take the namespace name and we can return an error. So now what we want to do is find all the secrets in that namespace that have the label we talked about. So let's do that. Client set, call, secrets are also called V1, I believe. Could be one, secrets. And now we need to give it a namespace that we want to list it in. Let me double check. Yes, namespace. And we want to list them, give the context. So we want these options. But now in the options, we want to say label selector and let me check the label. So we have, as I mentioned, we have two labels in each of the secrets. We're going to find the secret that has the class name set. You could be using the other label and it would be the same. Doesn't really matter that much, but okay. This way, the secrets that we will be getting are only secrets with that label. So we only will get secrets that Telegraph manages anyway. So obviously if error isn't new, just return the error and now we can iterate over each secret. Okay, so now let me explain how the secrets are set. The secret, I believe it was Telegraph config, that is zero. Yes, okay. So the way it works is that we have a single key called telegraph.conf, which is the exact configuration for Telegraph. We can even show this. I have this euros to the code plugin that allows the coding and encoding base 64 for an selected text, which is super useful for dealing with Kubernetes secrets because you have to do this a lot. So you can see this is the configuration. This was in the annotation. This was in the class. And Telegraph also injects gathering internal metrics of Telegraph by default and it's a setting that can be changed. But okay, this is the contents of the secret and the sidecar handler that we wanted to have in this, that we wanted to be passed. The secrets of data can generate that. There is a method called assembleconf, which takes a post that we don't have and takes the class data, which is the contents, which is this. This is class data. So now we need to do two things. First thing is we want to get the pod. Again, client set, core view on, pods in our namespace, want to get one. The name, we could use the labels from the secret. So you could do secret, okay, let's do this pod name, secret, labels, no, okay. That was what was missing, okay, secret labels, the labels, and we want to get the other label, label pod. So this is the pod name, actually it's just, so you want to get the pod and then options, we don't really have any option, we don't really care about any options. If there was an error, we just return it as well. And now we just need class data. And if we, we believe the site for method also has, okay, yes. So we can get the class data using a different, different, different structure that we have, different interface that we have, class data handler, and we just need to give it the class name, which is also class name, which is also a label. So we just need to, we just need to get it from the label as well. And I believe it also can return an error. So let's just make it a bit cleaner. Okay, so now we have the telegraph configuration, which is the string that we've seen in that secret, which is just concatenating all of the settings, validating this, this is a valid Toml file, which is the syntax for telegraph. So we have that, and now what we can do is, if the secret data, and let me the graph conf. Okay, this is, so this is the constant that we have, which is telegraph.conf, which is the key in the secret, that if this is nil or now that it's, now that we know it's not nil, you can just compare it as string. If it's not equal, then let's log something. So we actually use the logger for updating secret, and let's log something. So namespace is namespace, odd name is, okay, so name, let's log the name of the secret, name, odd name is, name and class, this class name, let's log this as well. This will be quite variable, but it's useful. So we can now set this data field to telegraph, or have to convert it to a byte array, and we set it. And now we can call the Kubernetes, let's call it one, secret, namespace. Actually, we could do this, because we were using this, we could just put this way. So update, and we call an update on the secret, we need context, the secret and empty options. So we're not interested in the result, but we're interested in the error. If it happens, we return it, and I think the error isn't any variable anyway. And if there were no changes, let's just log something so we can see that nothing happened. And let's take a look at the problem, see what are these errors are handling as well. Okay, yes, so I think right now, as soon as we clean up all of the potential errors, I think we're at the point where this code could actually work. So I'm going to build this while we're waiting. I can explain something about exactly how that sidecar, how telegraph operator works, the things that we're not touching today, but just so explain how telegraph operator works in general. Yeah, there was a question on the side about doing it as like a sidecar versus a daemon set. So that sounds perfect. Okay, right. So the reason we chose to do it as a sidecar container is that for some of the input plugins, so the means of getting the data, that data could be retrieved only from the same pod that the actual application is running in. So for Redis, this is using networking, so it would be possible to run it as a daemon set. We don't have the functionality in telegraph operator to do this, but it would be possible to do this, but we've chosen to use sidecar containers because telegraph isn't that large of a process. In some cases, it is easier to get more information could be gathered from within the same pod. And we found this to be also more manageable if there is a sidecar container, because there's also the matter of batching and then batching small sets of data. Because telegraph isn't always sending the data right away, but for example, if it's trying to send data to a database like an InflexDB database that's not available at this point, it will keep it in memory and send it later on. And then in that case, it is better to have more smaller instances of telegraph, I understand that this could be somewhat more resource, so using somewhat more resources, but we found this to be more reliable and potentially, well, we don't think we lose metrics, but I can imagine that this approach would potentially lose less metrics than trying to use a daemon set. So at this point, the operator only supports sidecar, but I wanted to explain, and maybe I'll just take a quick detail to do this right now, is how it works. So telegraph operator registers as a mutating webhook that gets a pod injector, meaning that whenever it gets registered, whenever a new pod is about to be created, such as by a stateful set of deployment, or any other way in which a pod could be created, which could also be third-party operators. Well, before that happens, telegraph operator receives an API call, which is called our admission request, where the type of request and the object gets passed. So for example, in our logic, we check if this is a delete, then we delete the secret that would be associated with this specific pod. So if someone, for example, deleted that deployment, and this triggered the deletion of a pod, then before this pod gets deleted, we also delete the secret of the telegraph configuration. In other cases, we call the site handler to check if it should be skipped, and then to inject, to generate the secret, then we add the secret to Kubernetes by API calls, and then we returned a modified pod spec. So at some point, we create the secret, and we also patch it with the updated pod definition. So that's what happens. Okay, so it built, so let me delete the deployment now, let me apply this, and we can, okay, I suppose, I suppose it was still, yes, it was still the old ones. Okay, so you can see right now, and this will happen every 10 seconds, that we get a message that it's not updating the secret because nothing changed really. So we can see the message that the updater was called, and then it checks every secret. I could, at this point, I could deploy some more examples. So we have, for developing purposes, we have a few examples. I deployed the Redis example, which is a stateful set, but let me also deploy a deployment which I believe is an ingress deployment. Yes, and that deployment has three pods because the number of replicas is set to three in that example. And we can see now that Telegraph operator tries updating the three secrets, one secret for each of these pods, and the fourth secret for Redis. So let's modify this, let's call this classes updated. And let's say, so, maybe I will show this as well. So in the app section, we can see that the Telegraph will be trying to send it to influxDB, which I may even have in that class, I don't really remember, but it's also writing to standard out. So if I were to do kubectl logs, Redis zero in the testing space and let's just take the last 20 lines, we can see that Telegraph was reporting a lot of metrics to standard output. By the way, this is line protocol, which is a protocol for sending metrics to influxDB and I believe some other applications will see use it. And that's just because we told Telegraph to log to standard out. But let's have it log to a log file. And now let's apply the changed files. Let's say, some examples classes updated. So let's, we've updated just this one app class. And if we take a look at nginx, nginx is using the default class, which is somewhere below, I believe. Or in for basic, I don't know what the default is. But it's using a different one. So now if I take a look at logs from Telegraph operator and we can skip not updating because we know there's a lot of things in there. As soon as this, the mounted volume that's based on the secret gets updated, Telegraph operator will pick it up. So we can see now that Telegraph operator picked it up. It said updating secret. And it's only updated the secret for redis because the one for nginx didn't change. So now we could try to get the logs for the Telegraph for redis. And let's just follow it. And in a bit, we should see a message saying that Telegraph stopped all the plugins, started all the plugins and we should no longer be seeing the outputs, assuming everything worked properly. It takes around one minute for the volume change to get propagated. But I'm quite sure that this has happened. So I'm not going to go and check the raw secret on my own. I'm assuming I didn't make any mistakes, I believe in that. But while this is happening, and hopefully we'll see the change soon. I'm going to go, okay, we've seen the change. So in a bit, we can check that there is a TMP log file in that port that is now growing and gets all the new log messages. So we can see that we have the first part that we wanted, which is I changed the class definition which triggered Telegraph operator to realize the change. We just faked how the realization of the change happened. And then Telegraph compared all the secrets and updated them. But it's a really bad idea to be updating every 10 seconds. So what we're going to do next, let me just, what we're going to do next is, let's try to use the FSNotify instead of a dummy method of just waiting. So let's just create the watcher. I believe it's new watcher, okay? And I'll explain the interface. If error is new then, it's an error. So a watcher on its own doesn't do a lot of things. It's just a mechanism. But what we need to do now is, we need to add all of the paths that should be monitored. So in the perfect world, this is what I would need to do. And also let me just watch around. Right. In the perfect world, this is what I would need to do. But because of the way in which secrets are handled, let's, okay, let's exit into the red spot and I'll explain. So first thing is, let's take a look at the log file. We can see it's happening. And I mean, it's there and I suppose it's growing. You can check it in a bit. But what I really wanted to show is how secrets are mounted as volumes because that determines how the monitoring should be happening. This is something we also had to understand for the monitoring of the telegraph config. So just monitoring the directory would not be enough because the way volumes get mapped is, there is a directory starting with dot dot. That is the current contents of the secret. So if I list this directory, I will see this is the telegraph.conf. Then we have dot dot data, which is a similar to the directory. And then we have the actual file linking to dot dot data. So if I would be trying to monitor the directory or just the specific files, it may not always be enough. And it's, so we're going to have to do something a bit more than that. So we're going to be listing all of the items in that directory. And again, if there's an error, let's just return it. And let's, and we're going to be iterating over this. This is more, this is less cloud native, this is more just file system, but still something we need to be doing. So what we want to do is we want to monitor a dot data, all of this, all of the semlings, but we want to skip all of the data folders and the parent and current director because this is not really interesting. So let's do this. Let's get the name. So I don't have to call the name method all the time. So if the name is dot dot data, then we're interested in that. Or if the name isn't a single dot, so we want to ignore that. And what we want to ignore is all of the other paths that start with double dots. So as prefix, okay, name doesn't have prefix two dots. So at least all paths except current, reviews, content and dot and two dots. So let's do that. So if this is the case, actually also let's log a bit more things. It's always better to do that than then logger info. I think do more check and let's log the directory. Also now, if this is the case, then let's generate a path by path join, class directory name. Let's also log this and then let's log this and can error equals watcher had this path. And if error isn't nil, put in the error. Okay, so now our watcher has all of the contents of the, all of the entire contents of the volume monitored. So we can get rid of our dummy monitor at this point. Monitor for changes. Since this is a Go routine, we'll have to handle errors internally as well. So our watcher uses a channel to send messages. So let's do this. We're going to be getting event and okay. And watcher events, we want to get an event from events. And actually what we will want to do is when a secret changes, there could be multiple changes. So for example, some sim links will be changed, created. What you'd really want to do is, I don't know if we'll be able to do this today, but I mean during this session, but what we would really want to do is batch all the changes. So right now we'll start with a simple version, but the next step is would be to batch it. So whenever we get this event, we could just call it on change, but it could also look event received. Okay. Now, so let's just check that this is okay, set to yes. I'm pretty sure I put something in here that happens during the live coding, but no, it should be, okay. It's this, yes, obviously. But okay, so, and right, okay. So I think we don't need to forward this, another thing. So I think that should be enough to get the events and this should trigger the changes later on. So let's build this. And in the meantime, let's try to do the batching. So the idea I have for batching is quite simple. What you really want is use another go routine, modify on change, which I'm going to implement right now and use a very simple mechanism. Maybe there could be a way to do this, but we don't have a lot of time anymore. So this would be, this would be a bad idea. So let's make two things in our structure. Let's make event count, which would be this and event channel, which would be a channel of Boolean because we need something. Okay. Okay. Right. So when this defaults to a zero and we're going to be using atomic, actually let's first check that this works. So I need to delete the deployment, apply the telegraph operator. Let's see that it started. Started. So now the last time we applied classes, it was the updated version. So to apply a change, we need to apply the original ones and let's keep the logs in here to see if anything happens. But in the meantime, I could continue with figuring out my code. So what we want to do is we want to get event, actually event channel. So we want to get, so what we want to do here is let's keep this, but instead of what we had there before, let's do something like this. This is how I often handle basic handing of synchronization. So every time we get an event, we're going to be increasing the event count by one. Also, we should create a channel, which is another thing, but we will send it through to the event channel. But obviously we also need to initialize the channel here. So event channel. So let's make. So we don't want to block, we don't want the go routine that gets the event from FS Notify tool to block your one. One of the things I may have not mentioned is FS Notify sends a notification about every single change. So an event describes the exact path and operation. So you could be getting multiple events. In this case, we only received one. We could see that it worked. So we could see that we have basic logic without the batching, but I've seen FS Notify send a lot of notifications in many cases. So it is a good idea to have the batching on top of it. Okay. So also let's log something, event received from the internal channel. And let's do this. Let's get the, okay, also let's keep somewhere the previous. Let's keep the previous values. And let's get the current values from the. Go check just a quick reminder that we are nearing the, our time here. So maybe we can leave that part in exercise to the viewer. Okay, right. So that's a really good idea. So this is the last thing that's needed. And it's just needed to produce the number of Kubernetes API calls. And just as I mentioned, like as soon as we finish writing this code and adding appropriate unit tests, this will obviously be made as a PR in the telegraph operator repository on GitHub. But yes, okay. Let's maybe let's just stop the coding session at this point. So let's start to summarize what we've done so far if you can just do a recap of what we've achieved. I think we've achieved quite a lot. I mean, I'm watching here. It's fascinating. We don't get a lot of sessions at this level in the show. So I'm very happy that we get an opportunity to watch how things are being made and not just how to use them. So if you could just summarize what we've achieved and maybe demonstrate it, that would be I think a good way to recap. Yes, okay. So what we've achieved is we are at the point where our Kubernetes operator can have a, can change settings in the telegraph operator classes. So the global settings for telegraph operator and how things should be monitored. So such as a typical case could be, I want to send my monitoring data to multiple source, to multiple outputs. I could just update my secret that's global to the cluster and we've arrived at the point where telegraph operator along with telegraph would have the hot reload of all of this. So I just update the secret and a few minutes later, all of the telegraph containers in my cluster would have the updated configuration, would start sending the data to different or multiple sources or would be handing the data differently. So I don't have to go and send secure signals or do other things to restart all of the telegraphs. That was the goal. Big win, big win. Yeah, definitely. Really nice. And if someone wants to follow up and watch the complete feature, where would they be able to do that? Yeah, so telegraph is on GitHub. Let me just open a browser and call it to my other screen. So we have a GitHub repository called telegraph operator. And as you can see, I was the last contributor here. So this is an open source project. We have, we're actively maintaining it. All of the, we have a CI in place. We have a release process in place. So the PR would be available in this repository. There's also a separate repository for telegraph itself, which is also actually maintained. And because it's a larger project, it has more commits and it has more things happening in here, but both are on GitHub and you can find them in our Influx data organization. Cool. If anyone in the audience has any questions on what we've seen, feel free to pop them in the chat or in Slack while we still have some time. I see that there was a question unrelated to what you were implementing, but just a general question about telegraph. So maybe we can address that. It was, so the first one we addressed already about a demon set versus a sidecar. There was a, just a general question about the comparison over Prometheus operator. You can address that as well. So compared to Prometheus operator, I suppose that the main difference is that telegraph operator was meant to, well, the primary goal for us is to gather the metrics into an Influx DB. And Influx DB has, well, with telegraph, it is possible to scrape Prometheus metrics and get them into Influx DB. And as Pat mentioned, surprisingly, as we influx data, we want data to be available there. Prometheus operator, I believe, is more about scraping into Prometheus. So that would be the main difference, I don't know. So if anyone's already using Influx DB, obviously they would want to check this out. And if they have any questions about whether they should choose Prometheus or Influx DB, where can they learn about that? I believe we have a lot of articles where we try to compare this. I'm not really sure if there is a single one I could point to, but I'm pretty sure that just going to a website and going to a blog. Yeah, I would go check out more information about the just about telegraph. And I think that will like, I think the big benefit to using this versus the Prometheus operator, like you can interact, telegraph supports Prometheus, so you can do that. But it's like, I think quite powerful, like you can do more. That's a good point, yeah. And one of the things that I think is interesting is Prometheus focuses on Prometheus metrics. And as we mentioned, telegraph has a lot of plugins. So I was able to get metrics from Redis by just using the Reddit input plugin. And as Pat mentioned, we have around 250 plugins. So for a lot of things that don't expose metrics as Prometheus metrics, telegraph can also most probably get those metrics and get the data for you. Got it, thanks. I do not see any more questions coming up and we are just at the top of the hour. So just another reminder, the link is on the screen if anyone wants to go check out telegraph operator and probably very soon a new pull request should pop up with some of the code that we've seen here. Someone wants to take a look at the complete product. Thank you, Wojciech and Tim. It was really unique stream today. I think we learned a lot and I wanna thank you, first of all. And just a quick reminder that we have KubeCon North America coming up if anyone is interested, it's going to be both in person and virtual, so go ahead and register for that. This has been Cloud Native Live, we're here every Wednesday. Come meet us again next Wednesday to learn about the latest in Cloud Native. So again, influx team, thank you very much. And thank you all of the viewers, yeah. Take care, bye. Thank you everyone.