 Good morning. Thanks for coming First of all a short introduction of myself. My name is some band egg I'm I'm one of the a singer co-founders from the project And my main responsibility now is doing community stuff and project organization and you can reach me on Twitter if you Want like to read that? Perhaps I'm not sure if you really want that so before I start Thank you to the has key guys for for setting this up for inviting me and help me getting these all these Visa stuff organized and everything which is necessary to come over to India. I like it very much and it's definitely not my last time over here Independent if you invite me again or not, but I will come back again So the a singer project before I start a quick poll so who knows what I think guys Okay, what's I 50% who knows what Nagius is? Okay, so we try to change this and in the next 30 minutes, okay What I think guys I think as an open-source monitoring tool So so a general approach is like we have a system which is extensible. We we check the availability We do notification And we provide data for reporting. That's our basic approach what we have So originally we we took the naga's code base in 2009 I think at this time it was naga stood to We took the code base into into I sing a one which is the fork we we are still running for years now But we started in 2013 to develop an independent core which is redeveloped from scratch So there is no line of code from the old naga's style in it and we have these I sing a two since 2014 now and This is a project driven Mostly by an international team So a lot of them are in Europe, but a couple of them are in the US in Africa and New Zealand Perhaps somebody in India. So if you want to join it's an open-source project. You can do so. We have a lot of work to do What we have as a stack is That perhaps you can see the pointer. We have the old I sing a classic This is what what what naga's was used to be like naga's with with the NDO utils We put them together and this is what we We used for years, but we had a several limitations with the old core There are a couple of reasons related to the software itself to the to the style how it was developed and also to the external Things like said we have still have these classic CGI's, you know these old naga's thing pretty shiny Thing we still have it and we're working on a couple of new interfaces. We have we have Reporting which is based on just for reports is an open-source reporting suit But today I will focus on I think I to and I think I went to so we are not Discontinuing I think I want so if you use this But we will not provide any major features for it So we focus on having security updates or if somebody sends in a patch So we will do it and we have releases after releases, but we're focusing on on our new products to bring them on the road So a short introduction what I think guys it monitors everything so you nearly can monitor everything It is an if it's able to be connected to a network you can monitor it or it's in sensor Whatever and we gathering the status out So if it's okay or not if it's reachable or not and we collect in performance data So every time we go to a server so the standard intervals about five minutes we go there We get is it okay, and how was the performance for example for an HTTP check? What was the response time? What we do is notifying using nearly every channel so default is email Which is the most easiest thing text messages popular the the most safest one alert is calling somebody So doing a voice call I'm doing a text-to-speech read the message and force him to say press 2 to to acknowledge These message because also text messages sometimes can take a couple of time until they reach the destination and And voice mails is the hardest thing to set up normally, but it's the best thing to to alert somebody So we consider dependencies So if somebody some service have relation to each other something like that and you can figure that we we able to handle it And we we handle events so we have something like event handler if you know an argues You know what an event handler is like for example if a web server goes down we can restart it executing a script We we check and forward logs Means all these all these information we gather we try to to provide it to other tools out there So we we provide locks using for example and galf output writing to lockstash or graylock We can write performance data to graphite or influx DB or open t-stp all these systems out there And also we provide data for SLA reporting in the database. So for now and we're supporting My sequel and postgres as a database back-end And also what you can see the on argues and all I think I was based on C We are working with C plus plus and boost which gives us the ability that we work on every Linux Unix, but also you can run isinger on a window system if you want to so it's not our primary platform But it it works really well. So perhaps you have some problems with the plugins on a window system. It could be But it in general it works and you can use also isinger as a agent on the system For all that basic checks out there So a lot of plugins for a singer we have a template library that's shipped with isinger So all these standard things this globe everything is configured You just have to use it, but you don't have to think about how we can configure the monitoring command or something like that So the current version is two three dot four it's it's out since April We have a lot of modules out there for puppet chef Ansible and we're working on salt stick So we discussed it yesterday evening So we hopefully get this on the road that we have salt stick support as well And if you want to play around we have we have regular and boxes where you can spin up the machine with all that Graphite stuff, so it's easy to try out So why is Nagus good? And the history of I think as all the I think are people which talked Nagus at this time. We were all huge Nagus fans. So we saw that Nagus has a big advantages But the development process was pretty slow. So let's say it was not existing. There was no development And therefore our our basic concept in August we really liked so it's very easy to monitor things with Nagus Just for me. It's a major reason why it's so popular because it's you just can write an easy plug-in. It's very easy It's a simple software stack So we don't have to set up a lot of cues and stuff and a lot of dependencies So it's very easy to get a first monitoring result We like that concept of active checks So you can do something like a monitoring agent on a system we send passive data You can do that, but I think that the most power you can have is really actively going on a system every minute Every second you can do that with a single tool and check if the status is okay or not and we have millions of plugins Okay, perhaps not really millions, but a lot of So the question okay, why a singer then if Nagus is so good one thing Nagus does not scale So if you have Nagus on a single box You see that one CPU is burning down the house and the other one is they do nothing because it's just a single simple loop And that's a big issue So there are tools out there and add-ons where you can spread the load a little bit or you can split up the configuration to a couple of nodes but this is a big issue in the in the basic concept of Nagus and This also comes to the problem if you send passive data or send commands to Nagus And he's under heavy load that he stops working. So these these load issue in Nagus is really a big thing. I Think a tool is multi-threaded. So it's it's using all the CPUs. Hopefully Load is distributed automatically So if you have so many checks you want to execute and then you can add more Checkers to the system and then they split up the configuration and and a couple of server checks the system So you can monitor in a second interval, but Caution here the plugin has to be good one. So if you start a 40 megabyte pearl monster every second We will not have a good weekend So benchmarking So benchmarking is it's not like saying something something all this bad, but just to give you give you an idea about so we see Online number one. I don't know if you can see this is I sing a one In the middle is a singer one plus game in gaming is a framework where you can distribute these these checkload to other nodes And in the back we have my singer to and we did this benchmark on a single box With 1 million services in a 60 second interval So there was it was it has been dummy checks, but for all of them So the the simulation was equal for everyone But we really good in in in doing this because we are a fork machine at the basic concept is like forking plug-ins And we can do that very good Another disadvantage in in August is adding modules is hard So if you for example want to use and you use tells want to write to the database It's like you have to download the files. Is it my mouse? No, it's yours You have to download in the utils. You have to configure make make install Sometimes you have problem with a little compiler and then you have to install it So there are packages available as well But adding new models because Nargo says he's event broker concept and it's really hard to add new models and all these things Would we solved in in having a modular architecture? So I think at the core This is what we what we ship and then we have all the features around it means we have I think I if you want to do Something you always need the middle of it But then you can decide on your own if you want to do checks on that a machine or if we want to write to the database Or if you want to provide a graphite So if you set up a mashup system you can have you can distribute the features to a couple of servers and let's say check on server a and Write to graphite on server B. They share the knowledge in between a zone, but I come to that later Everything is configurable using a C. I all the features stuff And it's very easy to add new things and so compared to that I'm going back to that example So this is like configuring into you do any utils on Argus and also perhaps I think I won It's like in I think I to is this thing a singer to feature enable IDO and that's it And then you have the database so don't have to take care of about Executing SQL scripts, whatever so it's just like enable the feature Internally it works like if you have an Apache website where you say sites enabled sites disabled It's just sets a soft link and then you restart it and it's it's the same for for graphite or other tools as well a Big issue with nagers is is clustering a distribution. So there is no contribute Distribution or clustering concept in Argus. You can do that on your own means you you can You can split up your configuration. You can copy over status data using SSH or SCP But there is no failover so you can use things like heartbeat to set these up and there's no application stag in there In I think that we have something like zones So we created a zone concept where you can put all the things in one zone But you can also make something like multi-tenancy where you have customer ABC and in in independent zones But have a master somewhere can see everything and How does it look like so for example you have a single box You can take all the features and copy them over to another box Which is just a high availability So it's independent if it's a Docker virtual machine or bare metal server You just say I install it twice then you set up an interconnection all the connection works using SSL and Certificates, and then you have an high availability scenario The other thing what you can do is like I mentioned before if you have two servers you can split up the features So we can say for example in this server. I have the database then you can write the database using the Icinga API so we tested a couple of it It's faster to to use the is internal Icinga communication writing to a database then using remote MySQL calls So if you want to write to the database It's best that you install Icinga there just enable idutils and write all the data from the zone to this Icinga machine and write it locally to the database. It's pretty much faster than right using remote MySQL and another feature You can have these multiple zones so we can have zone a so on B. They have they are totally independent independent so they have their their own brain they share the knowledge in between that zone and You can also do something more advanced like having a zone which is high availability You can have a zone and which is distributed in the zone with just a single box and equal can mash up all them together Which is if it comes to a setup like this, I would not say this is totally easy, but it's it's doable Severe it's it's advanced So another disadvantage is security so security in In August is let's say horrible nsca is And we still you need to use it in Icinga one So we also still support it because a lot of people use it, but please don't use it nsca is really crap Because nsca destroys your timestamp So if you send all data to the server and you have a situation that your network is down for half an hour And then the data comes to the central server then they and replace the timestamp with the current time Which means you can throw away all your sla data So please don't use nsca. It doesn't make sense It's better to to copy the performance files over with scp, which is the better way Nrpe has a couple of security issues all the time. It's the way they deal with this security So you can make it secure you can do something like Setting up scp or all these things you can do with with standard linux unix tools, but you have to do it on your own And I think are the cluster communication It's gonna happen using SSL. So we have a Bidirectional communication if you have a specific network zone You can connect into that zone or you can connect from the zone outside to your master So it's totally independent from which side you want to initiate the connection We have something like a bin lock retention means if you have a server somewhere else and the connection is broken for Let's say hours days weeks Then we relay all the data all the monitoring data later on and and write it to the database that you have it over there so because you you can disable all these features and The the memory footprint of the isinga core is about 10 megabytes at the moment With 100 checks So you can disable all the other features for example and just enable the checker And can bring isingar to your client So you can also use the isingar server as an isingar agent because it's so tiny You can just enable the checker you can now then Configure your your agent if you want to use these these kind of method Just put isingar there enable the checker feature and then you can configure every host with with your config management or whatever you want to use and then check them locally with the isingar agent and all there are Setting up the communication with a master server. So you still can use SNMP SSH and all the things out there So in the new version we bundle For Windows we bundle NS client. I don't know if you if you know it. It's it's NS client is a Windows program which enables you to really go into the details of Windows getting VMI getting performance counter So all the Windows information and getting these out and it's client is very popular to to do that And we will bundle that in the next version as well So it's just a simple wizard click-click and then it's it should be normally done Another big thing is the configuration limits. So with isingar 2 we are not compatible with the old isingar configuration and not compatible with the old nagers configuration as well, but this is many reasons We discussed that a long time because we know that that all the people were complaining about Mything things in the old configuration, but on the other hand, we knew that people were complaining if to have to write it new So people complaining anyway, and so we decided, okay, let's let's make it you so people We don't like it the first time but but we didn't want to make the old things again So I give you a couple of examples what we have so This is a typical service check Which you know from Nigus or isingar one You have a service and then you have host and every time you had a host you have to add these hosts And for example in a singer you can do that using a sample so you can add text to host And then let's say for example in this in this example apply service to every host Which has an address and is operating system is a Linux and therefore if you add a new host using your config management tool Whatever you just can add the host and then everything else all the templates are configured automatically So it's more a rule-based configuration Another example is that you can create combinations of it So you can say assign it but ignore it where where the server attribute is that is in test mode for example I want to check it but not in test mode Another example is these host groups thing I know that there are host groups and service groups in in isingar some people Don't use it But I think it's an interesting feature to to group specific servers and areas together But doing that manually could be a little bit hard and we can also do something like An object host group assigned So for example, we have an object host with the address with the check command and then we have these host group And with the name and then we say okay every host which matches this my sequel in a host name We add it to the host group. So wherever you add a host in your configuration. It's added to the system So one more thing so perhaps for the guys which are into into isingar Somebody has a clue what what this could be 12 minutes Somebody has an idea what it is. Yeah, it's it's a it's a time period based threshold So what is what we do we think conditionals to say is that server in that specific time period Then I said a specific value or for example load one load one minute intervals return 30 Otherwise, it's return 60 so you can work with conditionals. You can work with loops arrays and all that things So you would not be kind of example in every host service definition But perhaps you use this in a template and reuse it again. So this is very powerful if you want if you come to that level dealing with the configuration So just to summarize it's new Perhaps it takes you a couple of days to come into it, but you really really love it if you if you play around with it a little bit It's really really cool So leaving the the things you don't see normally leaving the core coming to the things you see So what you see is what you get from the Nagus perspective is hopefully not Because Okay, so you can do it. You don't have to But it's really not good But to be honest We use it as well. We have a different style We have fancy buttons and we have a couple of features like multi-select and multi commands, but it's the same It works if you have 20 services, it works pretty good But if it comes to that point we have multiple users Authentification dashboards and all the things the old interfaces is not ready for for enterprise approach Let's this say this way. So we also had and this isn't a singer sing So I'm laughing about our own tools as well We tried to put a mobile thing, but as you can see on the iPhone, it's Steve Jobs was still alive at this time So it's very old A Couple of issues with the CGI so CGI's need to parse status data So if you have a lot of services, you need to reparse the file again and again It's doing its reload and it could be really really slow if you have a couple of servers in it So executing commands remotely is very hard You can set up using as an SSH tunnel or something like that But it's very inflexible and normally you have to run the CGI on that server where you have the icing a core running On the other hand in our portfolio. We have the same issues So we have we have the old CGI's we have icing a web version one Which is which is not so bad, but it's hard to extend we used a lot of XML Which at this time seemed to be a good idea But now we know it's not a good idea because people never find the right place in the configuration to put things in so perhaps if you're into it you can handle it, but It wasn't in a lot of parts in the background. It was too complex and we had no inter unified interface so far and What we did now is that we have we're having a new interface. It's I think a web to it's very easy to extend So it's it's based on PHP We have a lot of authentication providers where you can connect to LDAP or internal DB or Active Directory, which is an elderb as well. We support Database or live status as an as an access API and it's responsive And the the technical foundation looks like that we have I think I web to is it's just the basic framework and Imagine all the other things like like perhaps like a WordPress plugin So the I singa is just the foundation which takes care of all the backend calls API and stuff like that and also monitoring Which we ship with and the documentation These are the only two modules which which coming delivered with I think a web to but They are modules so we can disable them you can kick them out It's just a directory and for us It was important to be to be totally independent from other things are coming in the future because we don't know for now And and what we have now is like we have a module for business process We have a module for graphite for PNP We have a module for Naguiz if you if you know that it's it's a visualization mapping tool for Naguiz and can do that and then I Hopefully have two three minutes to show you in a demo how this looks like Okay, so This is how I think about to looks so this is a landing dashboard you have In our design concept for now is that every time you open a detail for something for a service check Whatever it always comes in the column to the right so every time you do something and it opens a column So for example, if you have this critical check On the demo host you can see the plugin outputs You can see the details But you can go to the host as well or you can see your history which happened on this on this machine So we tried that that everything is maximum One click away so it could take a little bit in my in my system Oh, that's I set it up a new demo system yesterday evening. That was not a good idea but I'm for example You remember Check in in in Naguiz or I sing up one so we have to be true It's the same issue in our singer one as well if you want to reschedule a check It's like reschedule and then say now and then say do you really want it now and force now So you have to make 20 clicks to just executing a reschedule It's not rocket science, but it's just it's a single click so check now because I want to see it now I don't want to answer three more stupid questions So we try to focus that everything is really easy and it's it's really easy to Research something we have a global search which searches normally using the database So it's it's very fast if you have two thrill may learn service objects in there. We're using the index In my demo system. It's pretty small But you can for example search the documentation or you see the host group over there They have a host group with Linux servers with a couple of services. I can open them. I Can go to the detail See what's going on what the other services on this host are and Also can have a look on the host We have all these these classic parts like host groups and service groups We have an advanced filter thing where you can filter nearly every Attribute independent if there are if there are custom attributes or standard attributes come with the singer So you can filter all these things out if you want to and can Store them as a dashboard. You can create PDF or chase in or sees we in every view So you can also use just to be get to cut the chase and monitoring data out and What we also have is something like I go to the problems view something like a service grid and the service grid is an example that Show you the combination of hosts and services which probably have the same problem. It's sometimes It's very helpful if you if you not have a clue what's what's going on What perhaps you you assume that you have a problem with the storage? It's the easy method to get an overview and and see Perhaps there's a relation between specific outages and other hosts on the system Another thing we have It looks looks not very good in my demo system because it's too new is we have something like an event grid So mention the the the github landing page where you see the event grids You see the activity of users we have the same like here So you can go on a specific day or months and see If there's a lot so your more errors you have then it's more darker So you can open the detail as well and see all those things happening on a specific day, which is It's sometimes helpful looking for looking after things after a couple of times So looking on an error situation or an outage two three weeks later Perhaps you can see something like a pattern because some things happen every Monday every Tuesday And this is an easy an easy feature to do so We have also Something like a time-lining. We are we are trying to enhance this a little bit for now because at the moment I Just focus on the basic types The timeline is that you can based on specific time-line intervals go through your monitoring history Gather all the data and android into the bubbles that you can see the relation between Have there been a lot of notifications or a lot of comments or and did the people work more on these days? So you can have a little bit more than a data analysis On that things you can create your own dashboards if you want to Which is pretty easy just Add a link then you can add a new panel or a new dashboard and you have it and define it as a landing page. So Should be really easy to get a feeling about that So we tried to bring the reporting now We have in just the reports to the web interface and that current release We just have we just have the alert summary pot. I hopefully it works on my system Doesn't look so the alert summary is just a standard report comes with I think a web 2 Which gives you the top five recent alerts you have what's what's happening all the time you see the history You can scroll through it and see a graph. We will extend it in the future. So I think at the moment This is we have beta 3. So the release candidate will come at the end of May And then I hopefully we will be final in the middle of June and then we have five or six additional reports in that So you can you can absolutely use it for production. So it's ready But there are some missing buttons in the background and the user setting stuff But all that the monitoring things are ready and fine and we we know that a lot of people using it in production for months Now, but there are some issues. We have to fix now to say it's it's final Okay, just going back to my slides I'm over time. Okay Going to the community So one one third thing is we don't know Nothing about our users user base. So because you they use the packages clone the git. We don't know who is using I singa So yesterday I learned that snap deal is using it for example, which was pretty cool for us to hear We know a couple of companies we we put on here like Audi using it for the other whole environment They have about 30,000 servers using I singa few if you use it And if you're happy with it or perhaps you are unhappy with it, but let us know So on the website, we have something like a feedback form or a user form You can write it down because this is most valuable for us because we have no clue who's using I singa and that is something We're really interested in If you work with with plugins, so perhaps, you know, um, now just exchange for example or monitoring exchange These are platforms to exchange plugins but we have this issue for now that that Naga's enterprises kicks out every I singa plugin on their platform Because I think we're not friends anymore And therefore we created our own platform So we have our own platform sharing the plugins You can upload it and and what we have which is a big advantage you can sync them with with github or Mercurial, so if you have a project on github, it's just like putting in an yaml file in there Write what it is what category it is what software it is and then we re-sync your plugin every day So you don't have to take care because this is the worst thing using these these platforms You upload it once and after a week they are outdated. So you just sink your github project and we we update it again and again So we have a couple of events Locally, we have a nice singer came in Kuala Lumpur coming in June And we have one in Portland if somebody is going to puppet conf this year It's the day after puppet conf over there in Portland. So perhaps you can visit us there A partner of us provides training as well all in data Provide training in India. So they have oh, sorry They have the training online. So if you want to do a training or learn more about them So please check out their website And to to have a conclusion so our vision is that we we want to integrate all these things And it's just just an example of the things we integrate it now We really want to go a step forward. So having supporting things like salt stick working together with some And we don't want to reinvent the wheel again. So we don't want to develop a Logging framework. We don't want to develop a graphing framework We want to be open want to provide all these interfaces that we can work with them So our major target for for the new version. It's 2.4 coming in October. No, it's coming in November It's that we will a P5 everything so all the things we have now they are open You can select them from the database, but in I think a 2.4 We will have an a P5 API for everything and you can for example create dynamically host object objects You can create services you can delete them on the fly without restarting the I think a core And this is what we're working on now, but if you're using it now You're you don't have this situation situation like in Agus or I think a one and that you are blind during that restart process So if you have a huge environment and restart then you are normally could take a couple of minutes until the system is back And we don't have that issue. So we are not run creating the whole database. We just updating it and After the validation of the of the configuration we restart I think a and you don't you don't have a blind spot on it so Definitely download I think a tool and we prefer that you use the packages I think we are we are able in most of the distributions out there We also have a package server as well for where we put our Jenkins stuff in its packages.I think a dark So really we think your configuration. So if you if you using Nagas, whatever I think I think a to really rethink about Changing the way you configure your your hosts and and stop like writing 100 hosts down a big file So really try to create templates and apply them and and assign them. It's really powerful what you can do We have a pretty good documentation under Docs.Icinga.org so everything is listed down there every method everything But it takes too long to explain everything But but you really should rethink about how you do that approach because Monitoring should be should be a part of a bigger life cycle So if you have config management monitoring should be a part of it. It's it's no isolated thing It's not like I create services here and then open a ticket to somebody else and he creates a host So monitoring is very important But it has to be part of the of the big IT life cycle as well to to be automatically generated Install Icinga web 2 it's pretty easy to install It's just a PHP application put it in a folder We have an installation wizard where you can set up your active directory You can set up the database all that things it's you're guided through so it's it's really easy to set up And please give us feedback. So if you have getting any errors, or if you have a big wish list So please let us know and we try to figure out what we can do in the next versions So this is really important for us And though for now I'm through my slide take so thank you for the first time steps We have a couple of time for questions. Thank you so far Hey, do you have any automatic ways of taking a graphite graph and alerting on it I talking about if you can if you can monitor a graphite or so I have a lot of graphite data and I want Based on certain thresholds being reached Create like integrate graphite directly like if I have 10 graphs Just create thresholds on those 10 graphs and just I mean for example that you are alerting based on a graphite threshold Or graphite data because like it's just data, right? That's pretty easy. There's a plugin out there You can you can query the graphite AP which is pretty powerful And then on that based on that you can you can create a state and you can execute a notification command So it's out there. I don't have the name here But I know that the six systems thing querying the graphite plug in a graphite API and that's pretty easy to do But I was wondering if like you allow to say okay this You take this Graphite data point and everything under it. I just import Okay, I got it. No that we don't have that a moment. No So one thing for now is that we just provide the performance data and say we can write it to infox DB graphite So but then it's it's up to you what you do with it So we have a module that you can display the the related data in the web interface and can show the graphite information on the designated host and what we are thinking about in the future is having something like an automatic automatic learning So where we store the performance data somewhere and and based on the history and say okay Every Monday we have that peak or this this load pattern is different and compared to normal Mondays And then executing things on that so we don't have that at the moment But we are thinking about a solution solving that problem What level of integration do you have with Ansible presently? We have there's an Ansible cookbook you can download it and it's on Get I singa.org and there's an Ansible cookbook and it's supporting I singa I singa too I've never tried it personally, but They say it works, but there's always something saying it works, but now it should really work Hello, I Don't know where to look. Oh, sorry So what does the support you provide for virtualization and cloud or cloud stack or open stack? Okay reasons At the moment nothing So we can run we can run on an open stack or whatever But we don't have something at the moment very for example can query AWS and get the host out and monitor them automatically We don't have that because we want to do the API support first So that's our main target that we at the end of the year have these API and then have an external module For example querying open stack or querying AWS for specific host objects or specific services and then automatically add them To the core so for now what you can do I know that there are there a couple of modules out there where you can just query AWS for example then create config files So you can do that now But it's it's not integrated so it's just like it's a script out there So we just generate I singa config But we want to have that more into the detail that we we can actively query a cloud provider open ebola Open stack whatever and then generate the host or services. That's definitely one of the it's on the list. Definitely Thanks Welcome So I wanted to know that if you are making a SSH connection to all the host to get the data How you are planning to scale up? Because I feel like SSH connection will be having an overhead okay, so For that cluster this for a cluster method we have we don't using SSH or we're using SSL So and one way to to check a Linux or unique server is do that check by SSH So go on the server every five minutes for example in query data For our cluster environment using a this as a connection, but we keep it open And we don't reconnect every time so if the connection is broken we do the reconnect so we try to but Sorry, but we we try to keep the connection open so we don't reconnect to every agent out there One more question here is you showed a very good point of like, you know, you have a time-based monitoring But do you have like so basically all the monitoring systems are basically threshold best But do you have anything like you know trend-based monitoring? Okay, you are planning for any trend based no, but this is probably we have the answer with graphite This is something it's it's our roadmap. So we don't have that at the moment You can you can do that a little bit like using graphite using old windows algorithm So you can't do that, but we don't have any integrated solution, but this is definitely I think we cannot make it this year But this is on our list for for next year like like having these these trending things Storing the data for a specific time working with them and and find out what's wrong So but we don't have it for now You maintain your own database for the matrix or you are using the whisper database, which is being the primary database for graphite Yeah, so for example, we have we using my SQL Postgres storing the data But we don't store performance data for a long time. We just store events And if you write it for example to influx DB or graphite Then we just write it to to the IP address and port and and that's it So it's up to the graphite guys what they do with it So we're not taking care about what graphite does No passing data to graphite No, because the the easiest way so in our opinion was just directly write into graphite This is pretty easy it works and and one of our focus was to keep it reduced No, not setting up a lot of components and we didn't have any bad experience like writing directly to graphite so far Perhaps somebody else had a problem with that at this point we have a few thousands of like And I want 60 70,000 Nagy has checks at least or probably more okay So what's the best part to migrate to a king or two and we're using a king already so okay So there is something like a conversion script, but I would prefer not to use it because there is no intelligence in it It's just it just makes sure that Sorry, but your old crap is the same crap for the new version That's what it does so it transfers things because for example in Nagas We had we had contacts and I think that we have users and the conversion script does only does these things that it works again But it doesn't take care about thinking about what is your structure so how templates work? This is something you have to do on your own probably so if you have a config management and you have that intelligence there You definitely should try to to start the way of monitoring over there So it's just that monitoring is a part of that process But on the other side you have to take care on your own how your infrastructure is built on how template inherits Make sense so you can inherit multiple templates you can also override them So we can inherit everything and then on a specific point just say okay at this server I don't want these attribute so we can also delete attributes. It's very broken medically, but yeah We have no intelligence wizard which creates your good config This would be a one million product I think But I really so far we see people say okay I had a look on it and and they were yelling like oh I have to do everything From scratch, but after a couple of days and say okay, it's it's perfect I never want to go back because I don't configure every server I just use these apply assigned rules means you're reducing the configuration from 10,000 lines to 200 lines Which makes really sense to do so Sorry Rest API it's coming in November. This is the API. We want to provide chase an RPC rest RPC We are not 100% sure What you can do is for now you can get you can create the web interface get all the chasing doubt chasing data out But the API will come in November to get that Hey First to post or pardon. I didn't get the is it on the microphone. Can you hear me now? Yeah, okay So I heard you saying You have some road map to create a WCA pay calls and then get your configuration updated, right? Yeah, which is what we are doing right now. We are using icing guys works great So I don't really like the way it uses the CPU and memory, but it works really perfect But the thing what I was looking for is An event-driven Model to have the configuration updated rather you check it like every five minutes or ten minutes and then see what The change that you have in your infrastructure and then update your configuration For example that you that you watch a specific stream and say if some but something's coming up Then I create a host object It pushes The data and then something was listening to it and then I mean the you're icing a Server is listening to it and then making sure that the that instance is getting updated You have something in your roadmap I don't think so for now so our first target is that that we say okay, we provide an API and But but after that API thing the process looks different everywhere because you watch a stream other guys using a config management Somebody else is using Cisco works and dump this good works to get the switches in the monitoring So the the sources could be so big that we now first step say okay Let's provide the API that for example you write a python demon or Ruby Watch for that event and then just executing the API to create a host that that would be probably the first way to do it And then we will see you later on what is popular. So what can people? What would people see for a general approach to provide that what one thing with that? We are thinking about is that you can subscribe to two events also with the API that you can say you can Subscribe to a specific pattern if something happens on a specific host that you get an alert But we don't have these automatically generate thing So first step is API and then you can do it what you want one problem that you might need to look at is a false Positive that you get you with this method which I have a lot We're done with the question answers please take them outside to burn yes, and thank you burn and We have a container infrastructure. Thank you