 Okay, so that is the Etherpad link on your screen. So if you have any questions or any problem, just start typing the question. Hopefully, if somebody else knows the answer, feel free to start answering. If not, during the lapses, we'll go there and try to address each question while the lab is going on, okay? So please try to take advantage of Etherpad. So again, so agenda is presentation 15, 20 minutes, then the lab introduction will use it of the lab and hands on lab. We have only five laps, but those laps are divided into sublaps. And when you do, some of the laps will take good 15, 20 minutes. Some of the laps are very straightforward. We'll take like five minutes. So let's start with it. So everyone got a USB or everyone got already started copying on their laptop. Good. So as Paul mentioned, we work for Ericsson Cloud Manager project in Ericsson. And what Ericsson Cloud Manager is, it's a VNF management system across multi-site and multi-region. And we also have additional component that we use called Ericsson Cloud Analytics. And that product we use to get a performance data from OpenStack, which is across the site. So and in this particular product, we heavily rely on Cilometer. And as you see on the screen, what we do is like Ericsson Cloud Analytics will go in and probe those different sites, which has this OpenStack instances. So we have the same problem that common Cilometer user has in using a traditional Cilometer function, which is slowness, slowness of retrieving the data. So we manipulated that by reducing the time of how much data we are storing in the MongoDB on the back end, how many times we probe the data. And we kind of circumvent the current performance problem. So that triggered us to start looking at what are the alternatives. And luckily, like I believe two years ago, this Noki project was announced. So we started looking at that. And I'm not going to call ourselves very expert in Noki, but we have been playing with it from quite a bit long time and trying to learn. And this is our attempt to show you what we learned and what can you do with Noki. So traditional Cilometer architecture is like this. Now in here lab, how many people are already familiar with Cilometer and use Cilometer, okay? So very few amount of folks have used it. So I'll just cover the basic Cilometer architecture in aspect of what are the components involved and what does each of those components do. So it has something called polling agent. What polling agent does is it pulls the OpenStack services and builds the meter. Then it has a component called notification agent. What notification agent does is it listen for notification messages and convert it to the events and the sample. And then there is a collector, which is collect the event and sample created by polling and notification agent. Then there is a database back end, which is the data store. In the traditional Cilometer format, all of these mostly in the production environment, MongoDB is used for this particular one. And then there is API server, different sets of APIs to view the records stored by the collector service. And then there is a component for alarm, which is alarm evaluator and notifier component which evaluates and notifies any event based alarm or any threshold based alarm that you want to trigger. So these are the basic component. Now, since most of the folks here are not familiar with Cilometer. So Cilometer is a component that you'll have to install separately. You'll have to make sure you plan it right for it. So what is the back end database you plan out? How much space? How much is the retention ratio that you intend to store in the database? All that stuff you need to plan out ahead of time before going in production. But now looking at this architecture that there are two basic types of the data in the Cilometer. One is the event, another one is the metric. So event is the record event, which is like, for example, you boot up an instance, the start of the instance. That is an event from, it will notify the message bus, Cilometer API will send you a notification based on that. Or you delete an image. You add different volume, attach a volume, detach a volume. All these kinds of events. So Cilometer talks to the message bus and from there it retrieves those events. Now, there is something called matrix, which is measure the CPU users or memory users or data transfer IO or a network incoming and outgoing bytes. And then there are different set of database back end. As I mentioned earlier, MongoDB is most commonly used. Then you can also use MySQL, but for performance reason. Because think about it, right? How much data you store into a Cilometer, that matter. That depends on your inventory that you are storing in OpenStack, how many VMs you are creating, how many networks you are creating. And based on the message bus, every, there is a frequency interval in Cilometer, which will probe the data quite frequently. And as this probe the data, it keep on dumping it to the back end database. And it keep that data around for a long time. So you have to figure out that how you're gonna use your database, in a sense, how big it should be, how many, how many records you want to retain, all that stuff, all that comes as a part of the strategy. So now, as a part of Nokia, which is the additional component that everyone that has been introduced. And that use a Swift as a Nokia dispatcher back end. And that help in, because like in case of Swift, it is a component in OpenStack itself. You don't need to install a separate database component. If you are in your OpenStack architecture, if you are already using Swift, that will be your natural choice to store the Nokia back end database. So, looking, so just to let you know, so there is a lab one, just to retrieve a data from the standard Cilometer. And this will be a short lab, which will get you familiar with how the Cilometer back end CLI works, and what kind of the data it will retrieve. The major concentration in this lab that we spend bunch of time in learning Nokia, and that's where we are going to spend most of the time in this lab session as well. So the concentration area from our side will be Nokia. So what does it mean? So you saw the earlier architecture diagram and this diagram is a logical architecture diagram with Nokia. It is pretty much similar with what you have seen before. So the main difference is this particular highlighted area where you have a Nokia metering database and the event, it is separated out. If you notice, there are two separated database and then there are two separate APIs for that particular component. Also here, you'll notice there is the alarm component that we have is now changed to A. So again, we hope that we are pronouncing it correctly. It is AODH. It's an Irish name, it means fire. We googled around and searched that it is pronounced as A. Anyone Irish here in the room? No. So that's what we found out that it is called as, initially we were calling it as odd, but apparently as per Google it is called A. So I'm going to call it A now on in the lab. So that is the alarming component. So what happened is during the Liberty release and the Mitaka release, they started separating these components out. So there will be an alarming component from now on. If you start installing the latest version of Mitaka, the alarming component will be A. You cannot use a traditional kilometer alarm anymore. They are starting, the project is going on where they are kind of separating out whatever you had in the kilometer from alarming function point of view and they started moving it to the A component. So now why we are talking about this, right? All of this. What problem we are trying to solve? So there is issue with the cylinder today. The cylinder sample model is a free form metadata, which is flexible but it's too heavy for it. So if you look the way cylinder store the database in the database is it's kind of a big string. All the information it is collecting, it is basically dumping as a big string. It is good, but when you have a lot of samples in the database and you try to probe and retrieve it, it is very difficult job in aspect of the API. It's impact your performance. And because of this nature, it solves some of the use cases but not all use cases that is needed in the day-to-day performance data retrieval point of view. It is not solving all of them. And so again, so in the cylinder, we have currently the tuning problems like a large storage footprint, data intake optimization, and the query API performance issue. So this is to give you idea of the production environment like sometimes you issue a command, right? Or API to retrieve a data from a kilometer. And then you are just waiting for good amount of time. And sometimes your API will time out as well, where it will not retrieve any data for 300 seconds or something. And that's a problem. And in the production environment, you are relying on something like kilometer to collect the metering data. But if you cannot retrieve it and react based on those data, it is of no use. So that problem is getting solved by this Nokia component. So why we need Nokia, right? So it's fast and scalable. It's a metric retrieval is in the linear fashion in the order of one. So it resources index with the proper data type. And it basically writes the metric in the asynchronous fashion. And the biggest feature that Nokia has is this automatic roll up and the roll off of the measure. That means when it is collecting the data, it is keeping it. But based on your data retrieval policy or archive policy, it will decide when to roll up. So you can decide like, I want to keep five minutes of data. After it collect the data, it will roll up to that five minute or a day or a year. And it does it automatically at the same time. So when it retrieved data and it's roll up to five minutes, at the same time, the roll up for the day and roll up for a year also happened. So it's very flexible because of this roll up, your database is not huge. It keeps on rolling up and rolling off all the measures. And performance wise, it is literally like a day and night where in Nokia you can retrieve a data as quick as you can. Another thing is compared to Cilometer. Cilometer, as we said, it's a free form big data like a big string. In case of Nokia, it is nothing but it keep only two things, a data point and a time associated with that data point. That's all. It doesn't keep any other information in the database. But we'll go to the next slide where we'll talk about indexes, like how does it map them, the same data, what Cilometer is retrieving and if Nokia is only keeping these two data points, how does it compare? We'll talk about that in the next slide. In terms of the indexer driver, it is PostgreSQL or MySQL and that's what indexer will explain in the next slide. It has a storage driver. It can be file system-based. It could be a Swift-based. It could be a Safe or Inflex DB. You have a multiple choice there what you want to use. Nokia also supports the custom aggregation functions. So you can write your own aggregation types and method that you want to use in Nokia. So it is quite flexible. It works with or without Keystone. So if you want to bypass the OpenStack stack completely and want to use Nokia, you can use it. You can bypass Keystone and you can use just a basic authentication mechanism to retrieve that data. Another advantage of Nokia is it supports the multi-tenancy feature. So if I'm admin tenant or I'm a tenant-amol, I can only retrieve a data for that tenant-amol. I'll not get a data for the other tenant. So it has that multi-tenancy feature. So what are the key concepts? If you start looking at the Nokia documentation, you will come across all these keywords. So what are these keywords for? Resource. What is the resource means? So any type of cloud resource. In our case, from the OpenStack point of view, any instance or any volume that you create or any network that you create, any NIC port that you create is a resource. So it will have its own UUID in OpenStack. Metric. What is the metric? Metric identified by UUID and the combination of the name and the resource ID. So, for example, for the instance, instance could be one particular metric or a CPU utilization. That is another name that has given to a metric. So you can either use a UUID or you can use, like, for VM A and VM B, I want to get a CPU util. That's a keyword that they're using. CPU underscore util. I want to use, I want to retrieve a metric for the CPU utilization. So I can query based on that. I don't need to know, in that case, the UUID for that particular metric. Measures. So as I mentioned earlier, it just stored timestamp and a value. Only two data points are stored in the database. So that is what it is called measures. So all the APIs related with that, it's called measure in a silometer. Archive policy. So here is where it kind of difference between a very big and key difference compared to silometer, where what is the archive policy? It's admin defined data storage policy. And what you can do is it's composed of a time span and the level of the interval that must be kept while aggregating the data. So for example, right? I can retrieve a data like 12 points over one hour. That is one point per five minute. Or I can have one point every one hour over one day. That is a 24 point. So it automatically keep on rolling up as it retrieved data. So if I have like a 12 point over one hour, that means one point every five minutes. So it will keep track of the 300 seconds. That is the five minute data. It will, by default, it will use aggregation of mean. So what did that mean is the aggregation of data or funds and this is used to roll up the data. Also one point I missed while mentioning is granularity. So while retrieving the data, you have to, when you create any matrix in the no key, you have to specify granularity. So for example, you are creating alarm, right? And in order to trigger that particular alarm, you need to specify that what you are looking for, like every five seconds of data or every one minute of data, when do you want to, in what scenario you want to trigger, what threshold scenario you want to trigger the alarm? That's where this granularity comes in picture. And the retention ratio. So source data is not stored permanently. It's retained as per the data storage policy. So whatever policy we define as it keep on rolling up that data. So overall, the footprint of the database is very little compared to the traditional kilometer database. So here is the difference which I was trying to, so no key index, right? The no key main concept is to obtain a different metric data point, right? So resources are strongly types attribute, and metric record by the contact name. It could be CPU, UTIL, CPU, memory, or different types of name that they traditionally support. And it's loosely associated, the metrics can be cross aggregated for the resources. So what happened is if you see here, there is an indexer and there is a storage. So in case of indexer, it is keeping the resource instance. So it is keeping the record of actually what that resource belongs to, okay? And it will map to the storage. And it will basically link it to each other so that it will give you a reading of the data that you are trying to retrieve. So what do I mean by that? If you say like you run a no key API and you are trying to retrieve particular metric, at that time, if you want, for instance A, you want to retrieve CPU utilization metric. So what no key will do in that case is it will go first to the indexer. Try to find, okay, for this particular indexer, for this particular instance, for this particular resource, I need to get a CPU utilization. And then it will map based on what it is stored in the storage and retrieve that data for you. So basically from database concept point of view, as in Cilometer, you have free form of the data and it quite extends your operation just to retrieve one particular value when you have free form of the data. But in this case, since the index is there, the data retrieval is very faster. Also, if the resource type, if the resource type for no key is unknown, you can use generic type. So what is resource type? So when you run the lab, you will see that if you create an instance, Cilometer API will right away start retrieving the data and start putting that data based on your configuration. Let's assume the configuration is no key in the back end. Cilometer will start retrieving that data and automatically will start storing all the data related with that particular instance. And it will have, the list is quite big. It will have the network incoming, outgoing bytes. It will have memory. It will have CPU utilization, CPU, virtual CPU, all that information. And when it's stored in the database, it will store as a resource type. So in this case, for instance, there you will see a different set of resource type, CPU, VCPU, CPU tail, and so on. So essentially, like compared to Cilometer, Cilometer legacy storage captures full resolution data. Each data point has a timestamp, measurement, ID, resource, metadata, metric, and so on. But compared to that, in no key, you will see it store the aggregated data in the time series fashion, time series database. And each data point has a timestamp and a measurement. So resource metadata is stored separately and it is linked to the measurement. So as you saw in this particular diagram, it is stored separately and it is linked to that particular measurement. That's exactly what no key does. So this, all the no key related labs is we are trying to do in lab number two. It will help you to understand the, how no key does like a threshold alarm and how you can insert the measures if you or how no key get the metric list and all that stuff. So in the lab, keep it in mind that we are using the standard no key CLI. We are not using the API. What do I mean by that is we are not using the rest API. We are using the CLI which in the background will call all the API. So the lab two will be completely concentrated on no key. And A, so this is the alarm component. So a project is for the alarm code handling. It is separated out from the cylinder. Alarm feature of cylinder is deprecated after liberty release and completely removed in Mitaka. So as I mentioned earlier, if you try to install Mitaka from scratch, you will not see the alarm feature in the cylinder. It will, you will have to install this additional component called A and use that. A use cylinder or no key as its backend storage and each A service can scale horizontally based on the load. We can also trigger alarm based on the custom rules and that we can define. This is a very strong feature because there is a feature called composite alarm where you can use a different set of combination to trigger alarm. So that it is very powerful if you want to react based on the alarm. Also this diagram will show like how the A works. It has a alarm notifier evaluator same like what you had in the cylinder before. It is just, they have separated it out now and it is now, because they separated it out, it is very easy to scale just this particular alarming feature. And external system access the A through the service API. So we will have, there will be lab three and four with A. So when we will run alarm feature lab, it will use no key as a backend. So it's a combination of cylinder, no key and A, the lab three and four. You will use all of these components. And also there is the lab number five that we have added. Hopefully if you can finish one to four, lab five is a little bit for the advanced user where you will create a composite alarm or you will also use the web hooks to send the alarm to outbound to whatever URL you are specifying. So the conclusion, no key actually provides a significant performance improvement for retrieving the data compared to the cylinder. If you are regular cylinder user, you will see a significant difference while using no key. A separation from the core cylinder makes it quite a flexible architecture. And again, it doesn't matter what we are saying till you play with it, till you try out this thing, you'll not learn. So hands-on is the key. So let's get started with the lab. How many people have DevStack up and running right now? And how many people need help right now? Okay, so what we'll do is me and Paul will start walking around and try to address the problem. And let's take it from there. Once the people have DevStack up and running, we'll start with the lab. But if you already have DevStack up and running, please go start using the labs. Now, the lab, if you have USB drives, there is a PDF with each lab defined. So you can just start referring to that and start working on the lab. What I'll do is I'll come on the floor and start addressing the issues. How many people are okay now? Good. They have DevStack up and running and using lab? Okay. Let's... I just want to go over the lab real quick in a sense what we are doing in the lab. In the first lab, what we are doing is we are running bunch of silometer commands and getting out the different types of meter, the sample related with the meter. And that's all about it in the first lab. Nothing more than that. Just to get familiarity with how silometer is retrieving the data. In our case, in this lab, to fit everything in the DevStack, we put it in... We, instead of using MongoDB as a back-end, we use MySQL as a back-end because we are not storing that much amount of data and the main intention was to fit it into the DevStack virtual machine. So that's first lab. The second lab is where you change the configuration in the silometer.conf file and you add dispatcher as a no-key so that it will go to the no-key back-end. In order to take that in effect, ideally you just need to restart the silometer processes and the no-key processes. But in the lab, you would see that we'll tell you to reboot the DevStack only because not all people are familiar with the screen command. So rebooting is a quick choice of restarting all the processes. So that's why we are asking you to reboot. Once you reboot, at that time in the lab, you will see, let me go to the lab actually on the screen. Give me one second. So once you copy the silometer.conf and reboot, this is where we are actually calling out what is changing in the lab, what exactly we have changed. We changed the no-key, the dispatcher as no-key. What we found out while we were doing this test, the documentation of no-key was not that great, but on Tuesday night, the documentation was updated and some of the parameters that we have been using here are deprecated. The feature work has no issues there, but some of these parameters are deprecated. So for example, here on the screen, you see the dispatcher no-key, right? Now on it is, it doesn't honor that. It meter dispatcher no-key is good enough for it. So that is deprecated in Metaka. So we didn't know that again. Since we tested with this, I didn't wanted last-minute surprises if it doesn't work or my code base is not up to date. This code base is approximately two or three weeks old unless something is updated later on. We may not have it. I didn't wanted any surprises there. So anyway, so after you reboot and you try to run the same kilometer command, like a kilometer meter list, you should be getting this error, gone HTTP 4.10. That means it is not probing the database of the kilometer. Now, in this case, it is probing the databases, the no-key database. So you should get this error message. If you don't get this error message, do not even attempt to go further because all the rest of the lab need a no-key as a dispatcher. And if you don't get this error message, you will not have a no-key as a dispatcher. Then basically, we have different sets of no-key commands, like a resource list, metric list, and metric so, the details. So this is all for you to get a familiarity with the no-key. Also, in this lab, you will also learn that there is how the archive policy definition is there and what is the default setting in our case for that one. Then you will do basically get the measures out. So you will do a major show on the particular metric. Now, please note that the example that we have given here, in your case, it will be not the same. So that's how we, as long as you use a metric UID specific to your VM instance, it will be good. So again, now, let me know how many people have done with Lab 1 completely. Okay. And Lab 2? Okay. So before I go to Lab 3 and 4, let's check out, I'll come on the floor again, and if you guys need any help, we'll talk about that. Hi. One common problem that we are seeing across the board in the lab is Lab 2, where you can see, running a no-key command, you can see the resource list, but cannot see the metric list because it's a tenant-based, okay? So you are not sourcing in your OpenRC file correctly. Go to the home directory of the stack user and then do a source OpenRC and do no-key metric list. You will see all the data. If you are seeing the resource list and not able to see metric list, it is a tenant-based. Once you source in the proper tenant information, you will see it because you created a VM as admin tenant. And if your sourcing file is not correct or the parameter is not correct, you will not see a metric list for that. So just go to the home directory of stack, do a source OpenRC and do a no-key metric list. That's a common problem I've seen in the field right now. So we have five minutes left, actually. So are done. Actually, timing is done at 3.20. So a couple of questions came right. Is there a visualization or a GUI interface for this? So there is a project right now in DevStack. Actually, it is supported. I did not show the part of this lab, but there is something called Grafana through which you can see the graphical representation of all your data that you can do. It is not part of the horizon yet. They are pushing it for the Newton release. I don't know whether it's approved or not, but you should be able to see all the graphs and hopefully in the next release, like Newton release about that. Anyone who did lab five, I just want to guess, no. I would strongly suggest do a lab five where you'll gain experience with the composite alarm. If you are using alarming feature or intend to use alarming feature, that composite alarm start looking at that because that will open a kind of an entire new door for troubleshooting aspect of it and you can react based on that alarm. The example that we have in the lab is about VM AND. We are using an AND operator and the memory utilization more than 90%. But you can do a lot of combination there where you have a virtual app which has five different VM and you can use all those combination in the alarming feature and it will trigger alarm based on all that condition. So play with that. Then the lab five also has a webhook alarm where it will notify the other outbound webhook. So it shows you how to do that. That is a good one too. So one thing I'll tell you like some of the people who could not finish and still have the questions, I'm going to send an email out to everyone for... I will send an email out with the slides and everything but please send out an email to either me or Paul with any question that you have with openstack underscore submit Austin in the subject line and we will definitely respond to you. And I will upload the presentation on the Google Drive and send you an email and you can download it if you want. Okay? And thanks a lot for coming in here and do send an email if you have any questions. Thank you.