 Hello, everyone. Thanks for joining this session. We are going to add this here salameter and cloud kidney hockey and how one can leverage them to build a dynamic and agnose cloud monitoring building solution. Just not the line of the presentation that we're going to follow here today. We are going to start with an introduction with respect to cloud billing and how one can perform that and why one needs that in the different models that we have. And then we're going to showcase the stack that we the billing stack that we implemented with the open stack components salameter in our cloud kitty. And then of course we're going to finish with a conclusion with some takeaways. Just a brief introduction about about myself. I'm a cloud consultant and enthusiast. I am a PMC and committed for the patch cloud stack with this, which is an alternative project for open stack. But nonetheless, I have been contributing for open stack since 2018. And I'm a core reviewer and the next project team leader for cloud kitty. And they sponsor that is driving all of the extensions we are doing in open stack, not just in the billing aspect of it, but you know, even in neutral and keystone we have some other extensions that we're implementing. They sponsor the company that sponsoring all of these work is everywhere. And they are one of the leading cloud providers in Switzerland with more than 3000 business customers and over 20 years of experience delivering business critical platforms. And the interesting part is that because of the business they deliver, they have some very specific requirements, which pushes us to deploy a very complex and heterogeneous cloud computing environment. So just to give you guys some idea, at the same time, and because of some different business requirements, they have running cloud stack open stack v cloud and many other different systems that are used to build their cloud environment and to deliver their business. So just a brief introduction about the environments we are dealing with, they are heterogeneous, vast complex and dynamic. And that means we need to handle different types of devices different types of visualization systems different types of networking storage systems and of course, many, many different applications that are delivered to the users. And when it comes to cloud billing in the context that we are working on right now, basically the cloud billing systems systems, they are the ones that are collecting users data and these users that can come from storage network processing VPN users and many other different resources that we have in the cloud environment. And after the collection we have the data processing and rating phases. And these two steps that need to scale easily. That's one of the requirements we have when implementing such billing systems. We should also be able to add new metrics on the fly. And also of course, support different pricing for, for different conceptions of their same resources, and also support the transformation like whenever you are transforming somehow string data that you are capturing in some value or some of some sort. Of course, a change between different scales. So what are the options that we've, we found when we were looking for such a business such a billing system to fit all of those requirements, basically we found billing portals, and these portals they would create an abstraction on top of the cloud computing system. And they would connect a lot of applications and perform data collection rating billing voicing on board client onboarding and user management and so on and many other different features you know like some predictive analysis of the environment some forecasting for the resource consumption and so on. So after analyzing these different systems, why didn't the client that we are working with adopted a code portal system well basically because those are our normally proprietary solutions, at least we did not find any good open source solution out there. We are easily customizable. And so, you know, we need to add new metrics on the fly we don't actually when we started two years ago we didn't know all of the metrics that we are using today. So therefore, the proprietary systems they did not feel that easy to extend it to customize. And when we say customization here we're not talking about you know changing icons and CSS and so on. We are actually talking about the type of metric that the system monitors stores and then you know use uses for billing. And also you know whenever integrating a new system or a new device or a new application. So for them they have the supportive variety of different application systems, but nonetheless whenever we would need to add a new one for instance we would need to raise a ticket to discuss, you know how this implementation would be executed and the cost and so on. So that would make them a little bit slower to to support new metrics and new integrations that they need. One alternative to cloud portals is something called monetary quota, and that's a feature that I have seen applied only so far at least only on Apache cloud stack. The idea is that you limit the user like quota you limit the use of computing resources for a user based on a monetary value so instead of limiting the amount of virtual machines for instance that the user can deploy. You actually assign the user a value like a credit, and then the user can consume that credit by using and allocating virtual machines and each one of the resource that the user allocates has somehow a monetary value that's attached to it. And well for us for this specific client that did not work, because the model this type of model is intended for private cloud use. And because the client is actually a public cloud model and we need to something to mimic the pay as you go, which is actually the public cloud model that would not fit and also you know open stack did not have these implementation built in. As far as I know at least. And last, the last model that we were discussing and we evaluated is the idea of rating. So the rating is just about, you know, assigning a monetary value, a price for for for the consumption of computing resource or I called computing resource for the client and in a given time. And then these applications they would normally generate reports you know, explain showing the amount of resource that was consumed and the monetary value for those resources that the user consume that that's the model we adopted, and it fits nicely in the public cloud model. And open stack has a code key that has a component called called called kitty, which is a rating as a size module. So it fits nicely the requirements we were working with the, the chain that we implemented. And of course here we are for instance hiding the parts that after called kitty, we have like an ERP, another system of the company that's kept getting these resources that were rated that that were that had a monetary value assigned to them, and then you know generate a rating and send to the client. But basically, we have the cloud environment with all of the different resources that once needs needs to handle like identities, user connections, object storage, computing resources and so on. All of those are monitored by in different systems as well for instance the cloud is a different code computing orchestration then open stack. And then we have silo meter monitoring gathering data there and persistent in your key. And after the data is persisted in your kids consumed by quality kitty so we can rate it so we can assign a monitor value to it. That's a very simple stack all of the components they are open source which is interesting and we can extend quite easily. Of course we implemented this but you know this is not just what we have production because as soon as we started implementing it, we found some limitations. So, for instance, silo meter silo meter was not able to create new metrics on the fly all of the metrics that silo meter had built in they were hard coded meaning. There is a Python code that is executed and that collects and handles the metric. That's the problem because we wanted to be able to generate new metrics, meaning collect data from different API's and from new API's in runtime. And that was not possible when we started working with silo meter so to overcome that we implement we created a new subsystem and new feature in in silo meter, which is called dynamic posters. The idea is that one can define the posters the agents gathering data in silo meter via email. So basically you declare the open the open stack API that you want to call the parameters you want to retrieve and how you want to process them. And then you store them in the back end and in our case the back end is not key. So that's the first extension with it and then we noticed that okay we want to monitor not just open stack resources as I explained to you guys at the beginning the sponsor has you know, a very heterogeneous called environment, which means that we need to monitor different, we need to be able to monitor different systems at the same time, and ideally with the same billing pipeline that we were building. So we extended side the dynamic posters in silo meter that we created even further, meaning we enabled the dynamic posters to monitor any PI that we want as long as they come, they, they expose a JSON format. I put an asterisk here in the implementation because silo meter had an integration with the radios gateway before with S3 protocol. But as the hard coded poster it was something you know written in Python it could not actually easily extend. And when I say easily extend here I'm talking about you know the operators, because somebody from the operating that's operating this whole infrastructure they might not be that familiar with Python, or might not be that familiar to extend this type of modules. And therefore, for them it's way easier to configure a new poster. That's what we introduced with a dynamic poster using YAML for instance then then coding the poster with Python. And by delivering these two implementations, we are able to monitor basically any type of API that responds JSON with with silo meter right now, even if the API is not an open stack. And, you know if the API is requires credentials requires authentication for instance silo meter has a built-in integration with with Barbican that we also introduce it. So, all of the credentials are safely stored in Barbican and retrieved by silo meter and then use it to connect to the system that we are monitoring. So, it's a quite solid and secure monitoring chain that we extended silo meter to have and to support. After silo meter we started noticing, because we put the system production and then you know everything was nice until it was not. So, for instance, we started noticing that in your key we was not returning the right version for the attributes of the monitor the results by right version of the attribute here is I mean for instance, in the middle of a virtual machine, it's stored in your key as an attribute. If the flavor changes in the middle of the month, for instance, and if we do a reprocessing for that month called in your key was always returning the latest metadata that was stored there. So, that was the problem, because then instead of using the correct value for the rating, we were using the wrong one the wrong information to define the price. So we had to fix that for in hockey as well. And after fixing that and of course we will continue moving on and with the run the system up and running. We start noticing some slowdowns in hockey and that was happening because of the database scheme that was implemented to it. So we had to implement to improve and to optimize some of the office database schema as well to support the growing data sets that we have. And last but not least, we also extended called kitty. So we needed to introduce the support to the rate xx aggregation methods in your in called kitty, because called kid was not properly using them. So the rate xx, these are some options in hockey that allows you to calculate the rate between two different to two data points in time. And called kid was not actually executing that use of the API properly when these aggregation methods were being used then we fix that as well. And last but not least the usage reports called kid was had a fixed usage report model meaning it would only return the resource ID quantity of the resource that was used in hours and then the price. So for our use case we wanted to be able to retrieve for instance the metadata the attributes that are stored there in called kitty as well. So we extended this API to allow integrators and other people consuming the called API to retrieve the metadata that that is stored to it as well. So whenever you retrieve for us whenever we are retrieving data from called kitty, we are able to retrieve not just the price and the quantity but also the data attached to that to those rating formation and then we can generate more meaningful reports there. In conclusion, we were able to build a very solid and open source building pipeline with silo meter in hockey and called kitty. And that's interesting because we are able to monitor and build all of the cloud resources. So it's not just about open stack resources. So everything else that we have in the cloud environment and that's supported by this pipeline. We have a limitation right now. That's worth mentioning the extensions that we did in silo meter they are not able to to monitor XML API so for the instance the V cloud that we have right now. The data is captured by some scripts and then persisted in hockey, but after the disease in hockey called kid is rates them as we rated any other resources. So that's one of the last extensions that are missing in the dynamic poster subsystem that we introduce in silo meter. And this setup is stable and has been in solid and it has been running for over two years already. And we keep improving and working with it and not all of the implementation but most of them that we did they are already in the upstream. And they were already released for instance silo meter all of the implementations we did in silo meter so far. They were already released in victoria release of open stack in hockey some of the pull requests for measures some of them are still open. We do know that in hockey is in like a gray area right now it's not part of open stack. It's not a component from open stack but it's quite heavily used and there is a movement to somehow bring this type of system back to the community. But so far when we when we fix it, those limitations we propose the fixes back to the young project in GitHub. And in called kid that is a think one extra pay pull request measure request that's still open there but you know it's it's coming at least probably in wallaby release. And for the future that's, we are only missing one thing one step to be able to build a full blown called portal with open source. And that's what we are working on right now. So basically we have developed a customer onboarding and management portal that enables us you know to do the user onboarding to manage the users and to integrate different systems in the in the in the in the cloud environment and to provide that unified unified view of the cloud computing environment for the end user. We are right now testing and validating it internally and the system will be open sourcing the coming months. And that's basically all of my presentation. Thank you guys for watching it and if you have any further inquiries questions. You can just ping me and send me an email in my private mail account and I'll be happy to reply to you back. Thank you. Bye.