 Hello everyone Yeah, I will introduce myself my name is Ioannicius and I will spend some time with me talking about the insights and I hope you will enjoy at least a bit of it. If not, no internet is free You can also we are not locking the doors. So but I hope You will actually get something out. Also one important thing the There's a lot of slippery slopes and not only slopes out there tested on myself So be careful and not not fell not fall as I did So that was the most important thing you should learn about this talk And I guess we can just go ahead and get started with with the content. So one warning This presentation contains some subliminal messages put problem is probably put a lot of reclamo. So Watch for that and I will ask you at the end. Like what what was the content? So be prepared. So a bit about me I am this little guy here. I act as a tech lead for initiative called connected customer experience and Yeah, we are trying to improve the support ability of our products based on the health data that we are getting from the clusters That are running off the redhead products. So yeah, I'm trying to to help out a bit of technical part there So the problem domain that we are going to talk about is how we can actually help running the software When you don't have direct access to the systems and the only thing that you have is Some and usually multiple ways how to gather the information about remote hosts So we need to somehow be able to get and and process and Deal with the data that we get and we of course have also a lot of formats So you can think about different formats of the log messages different formats for the configuration of the software because you know That's fun creating new formats so For example, like what kind of data we are talking about when you run your software You know, you might have some minimal requirements like when you have 200 megabytes for running Opens the core opiations or whatever. You'll probably not succeed. So, you know, one thing is that just knowing where the software is running Also, knowing how the software is configured has Or can have a lot of impact in terms of like if the software will run correctly or not and Of course, all the software should produce some Information about it's run. So you might know bar lock or sure now deal, whatever it is for collecting the the locking information And of course, like there are many other other things we can watch for So what I would like to talk about is one project one particular project that is open source And it's called inside score It's available at github and it's the as the name says it's the Sitting at the core of the redhead product called insights redhead insights that we are building to help our customers and give them advices about the How they run our software? So this will be a presentation about how this software like the core of it works and It will also have some we will experience some advantage with or adventure Let's say with the live demo, which is always fun So this is the one of the tools that can help you actually with Dealing with the data that we are getting from the customers and Analyzing that so some of the properties is that we try for it to be versatile So what I will be showing is running this tool against life system So like I don't need to kind of gather any information in terms of like copying files or at the disk and so on But we can also Run the tool against the data that were collected from remote machines So think about source reports as one of the examples. How many of you know source report? Okay, so You are familiar with that with those that don't it's basically a tool that allows to collect as much health data information about the system to be shared when somebody else tries to help you Yeah, and this is not just a set of like base scripts or some set of regular expressions with set That you could do it do as well We actually using it as I mentioned as the core of multiple production systems So we try to be as robust as possible and at the same time We also need to minimize the set of requirements or dependencies for the software so that actually we can run it and Make it as a as a dependency by itself So we don't want to download the whole in the internet when you are installing this thing Yeah So the purposes Are We can parse the raw files with it We can define data model on top of these files these data and then we can either Define rules or extract additional data so that we can do for the processing so the idea is that When you can have multiple sources about the information you can have the life life data I can run some commands to get like information about the system or you can have the srs reports or you can have the insights archives themselves We have additional way how to collect data about the system So the idea is we have many ways how to collect the data But we want to have a single model that describes the this data and allows us to treat it in a uniform way So once we have the data Model as I mentioned we have multiple sources of this data The more model approach also is to use the prometheus for for matrix And we are also using insights to be able to work with that so We can show this example with for example looking at the Information about the memory. So this is will be with this will be the fun part about the live demo so I'm switching To my Terminal hopefully you see it and I will just run Python so one thing that I've not mentioned the inside score is written in Python and Basically when I talk about the modeling the data, you know We are modeling it using the the Python objects, and we can then use it this way So the first thing you need to do is From insights This is for The use case when I'm actually using just the Python Shell or I Python to export the data. I wouldn't show how we can actually Use it for defining some rules. So what I'm interested in is the memory memory information about the system so the inside itself defines a lot of different parsers and Models already that allows you to do this. So we don't need to parse the memory information of the system on our own But instead we can use already what's already there. So from the insights We have parsers many parsers defined Memory info is one of those and I've already looked in the documentation that I'm interested in the man for class And I have invalid syntax Import okay, so I imported my class like nothing happened yet What I will do now is run the gathering phase of the insights that will happen just at this computer and Store it in some some data. So That's why I imported the run method in the first place. I will run that Since it's running real life gathering of the data it takes few seconds And then I can ask these data object to get be the mem info object of the main menu for class Okay, and When I try to search about the or look for the total Attribute or parameter on this object. I actually got the number For those that don't see below. This is the total number of What is bytes kilobytes megabytes? I think I think it's bytes actually Of this particular machine so without needing to do anything else I have the information about the system and we can use it for for doing some some interesting stuff with that so That was just like short example on how these Models parsers inside score itself can be used to deal with the locomachine and In the further examples, we also will look how this can be used against for example the source reports So one of the main reasons like why the project was created Is actually to be able to define some rules about the system So when I have the memory information and I have software that requires Some memory to run successfully you can combine this information and when you codify it allow the customers or users to run it You can give them a unified way how how you tell them whether they are good or not in terms of running the software so yeah, they're defined on top of the model and Logic can indicate the potential problem sometimes it might not be problem itself, but it still may Suggest improvement so now you might be running the software in some Misconfigured way and if there is somebody that knows how to configure it properly. This is the tool that allows to Automate this thing. So yeah, sometimes, you know people Use your stuff. It somehow works, but it's not Done in a way that was meant in the first place. So, you know, like you probably can't get into into the locker but Maybe not designed that way and you can be more effective. So let's let's take a look at Seeing how the inside core can be used to actually parse the Some configuration files. So that's another thing that is pretty neat about the Inside score is that defines by itself a set of parses on top of the configuration files That allow you to work with this with this thing more in more programmatic way. So one thing that I will do is looking at the configuration of the HDPD I've actually Configure it here So just to show you my HDPD The I can't speak and type at the same time So I have here some configuration about the Performance characteristics and I set up the max request workers for the Apache to be five So let's see how I could get to this information using the inside score and the idea is that no if you have some total amount of memory to use For let's say for the HDPD Then it also means that you have like if one worker in the HDPD has Requirements minimal requirements for for some memory you can calculate Was the maximum amount of workers that is still suitable for this machine? And if it exceeds like if I configure the number too high here I Would expect that something could tell me that you are doing something wrong either increase your memory or Lower the the max request workers. So let's see first like how I can can treat the HDPD HDPD config files one thing that you can notice that like the the configuration of the Apache or the HDPD is composed from many files. So, you know, you could grab the things like that but We can probably do better. So what I will do in this example is using the HDPD parser and the three representation of the data. So from insights, I will not explain in this talk what combiners mean but I can Give you some information later import The gets three method actually gives me the three information. So if I do HDPD I get three I Actually got the content of all the configuration for my HDPD and I can for example And let me just import also some querying helpers and What I can do is in the HDPD find the configuration or configuration key that Contains Workers, okay, so this Loaded the data looked at all the configuration of the HDPD and Somehow found that this particular key has the value five So in this like it just allowed me to explore this without me having to know exactly. So this is also kind of helpful in exploring the data And I see that I'm moving too slowly. So I was Which into the past Face so bear with me So, yeah, this is the the thing that I was showing you we have multiple ways how to how to pass this thing And it also extendable for for more information. So What we can do once we have the Parts are defined is defined a rule Which in this case will look and that's what I mentioned before we look at the memory information of the of the Machine, we look at the HTTP configuration and compare if the size of the workers is actually matching the required Required amount and yeah, one thing that I have not shown is that we can actually run this against Some SSR support as well. So if I go and run this particular file, which is the FHC must mark max workers and I run it against Some data source report with some HDPD Actually see that the rule max workers check has failed because max workers is set to six and the recommended Value is free like we could have more Formatting about the message but you get you get the Point there is also another thing that the insides do is actually Defining the dependencies and we can handle the case where the data or some rules that we are defining are not actually present in the file so let's say if I run it against the Archive that has no HDPD installed. I know I don't get exceptions I get information that some missing there were some missing dependencies. So you can actually have these rules defined for multiple software so Yeah, that's that's what was covered with but this slide and I also showed this example Yeah, so another thing that we do is sometimes writing the rules itself is not the kind of the most efficient way to do but we can also take the data from the archives and do some AI with that and Sometimes the AI means just a bunch of it, but after working with this stuff for some time I'm less skeptical because I actually seen some success with applying machine learning techniques on this data So I think next week or next year will have more to tell talk about this part as well But the first thing before we can actually do this kind of processing is actually extracting the data from the archives From the raw format into some format that we can actually plug into the machine learning algorithms to get some some additional Information so I will not show the demo because we don't have that much time for that But the idea is basically being able to extract some facts and then plug it into machine learning Mechanisms machinery so And that's brings me to what actually we do in the insights right now So insides has been around for some time already over two years I think maybe more and they were focusing on the on the rail on the operating system case And what we are doing right now is actually applying these techniques at the open shift space and this Brings new challenges. We have things that are progressing more in the open shift world We have more health information already and we can still leverage the thing that we already had to deal with this data and actually provide the similar Yeah, the similar Suggestions as we do for the Operating system row So I will not go to the examples But at the end of the day what we do is actually try to search for problems and give hints So this says that this might be how windows used to do that sometimes maybe not not anymore So we try to do like the result of our work should be instead of the sleep Actually do real work and then getting the real real suggestions out of that so And we are hiring so if you are interested, please Let us know and that's gives us to the Q&A session. So please if you have any questions now, it's your time No, and Yeah, so the question is if we are running only against source reports and the answer is no as I mentioned the source report is one of the most Like contain the most of the information But sometimes you don't want to like send everything and we are for example, you can define passes for the insights archives In the open shift world world, you might be familiar with the mass gather tool Which is equivalent of the source report but for the open shift and we have Define the passes for that as well and we also can load for example data from the promity you so like we can still Any other question Yeah, so the insights client that's the the client tool that we are actually using for collecting the regular data about About the system and the question was if it's running only regularly or if we can even collect ad hoc information or run On demand so right now we are running periodically every night and what can we kind of influence is the For example, when we are looking for the new error message in logs, we can kind of extend that so that's that's the way how we can Influence what gets collected, but we are not running it as a demon that we would be triggering on demand That like we might turn that at some point, but but we didn't Yeah, right now we are focusing on the cluster because we if you have broken question It doesn't matter even to think about the application health, but once we figure out that part I expect that we would be moving towards the layer above that Yeah, so we as a team Yeah, the question is if like based on the information if we can do the remediations only like from the cloud side Or if there is intent to also be able to do something locally So we as a team are focusing on the cloud integration and the cloud remediations So if there is any of need for doing this or being able to leverage this even like an on on premise without connecting It would probably need to be more effort of the product team using that but the tool itself is pretty straightforward Simple to use and these integrations can be used without any cloud integration as well So like that that was the purpose of this talk as well is to show that you don't need almost anything You don't need to be connected through the cloud. It had come to be able to use inside score and And I guess that actually took us to the end of the presentation or we have one one more minute So one quick question. Yeah, I have one question. What was the what was the message that was there? hidden somehow Space max and the second part was Rocks, okay, so so you at least know my favorite editor and I had to put it somehow into the message So this is the end of the presentation. So thanks all for coming. I hope you enjoyed it