 I'm here to talk about Osmocom, KPI, counters, rate counters, stat items and stats, KPI for anyone who doesn't know is just or means key performance indicators in this fancy talk for counters, rate counters, so everything we can measure. So in the history of the Osmocom project we've had counters or some sort of statistics for a long time. It began actually even before we had Libosmo Core in OpenBSC in 2009 where we had the first type of counters, the Osmo counters. Those were within OpenBSC periodically written to a database which caused some nice issues later on when we had, especially at the CCC, the event database grew so long and had all the counters that were, well, appended there every minute. Yes, the early integration of counters, so that was keeping track of counters and just getting those counters out that we could look at them and visualize them was just some simple script that even copied the database and dumped the stuff we needed and then, well, did, well, visualize them with RD tool or something else. So that was the history. Today we have three different implementations for counters to choose from. They do different things. Only two should be used now if you're looking at adding counters. So white noise space here. You can use them to track rate of events over time, like location updates, accepts or requests and rejects and calls attempted, calls completed, and then, of course, you can visualize those to check whether your network is healthy or not. You can track also momentary values that change in a discrete manner, so not just rate counters that are monotonically increasing, but like having currently active calls or the delay of some connection, how long it takes to act, acknowledge a packet. We can query all of these different counters through the VTY interface, so just show them quick and dirty. We can also query them through the control interface, which is a bit more programmatic approach to create from script. And we also have a stats reporter and implementation to basically send all these counters to StetsD and then use whatever you want to visualize them. Yes, the OsmoCom, or at least the OsmoCom GSM project, we have quite a few different counters, so tracking calls, SMS location updates, or most of the signaling messages, what has been attempted, what has been completed, also for GPRS Attach, PDP context activation, the whole bunch, basically, and also data sent for GPRS in both ways. Yes, and many more, like reconnects on the A side or the A-Bus side with OML, RSL. Yes, if you want to know within a certain project which kind of counters there are, then you can look at the documentation, the counters are automatically or periodically regenerated into the documentation to give you an overview of what is there, or you can use show ASCII doc, VTY command to basically get the same list that the documentation generation does, just maybe it is more current. Okay, so let's get to the first type of counter. It's, we call it Osmo counter, very intuitive. No, that's intuitive. I'm getting to the pun with the next counter, which really is counter intuitively named, I think. So that's the oldest implementation has been there since 2009, in various stages. So first it was just some fields in the struct that get added upon and then it became a little bit more generalized. So you have names for a counter and it's not just it, we keep track of them in a central location so you can actually output all the counters at once and don't have to know where they are and what their name is. It's a very simple interface, you can increment a counter, decrement it, you can get it, reset it and later on you can also get the difference which will automatically reset it, which was needed when the stats deintegration got added. It's still used today for things like active calls or some reconnects in the MSC and BSC, but it's not really used that much, so maybe five to ten counters or so, I think in the Osmo-com GSM world in general. And even today I looked in Osmo-MSC, it is synced, so only these counters are synced to the database that we still have to keep track of SMS every minute and you can disable it and I would strongly advise you or maybe we want to reverse the logic if not remove it altogether. So with the option minus C you can disable it, maybe we want to have an option to explicitly turn it on because I don't know, just noticed that when I looked over the code. So yeah, this how you use it, Osmo counter alloc with the name, then when the call is established, for example you increment it, afterwards you decrement it and then with the VTY interface show stats will show the ungroup counters, those are the Osmo counters and it will show the MSC active calls is currently at three. So the next type of counter is the rate counters, those were an improvement to the Osmo counters introduced in 2010 and somehow counter-intuitively it is missing the Osmo prefix even today. So yeah, when I started writing the slides I put Osmo counters, the Osmo step items and Osmo rate counters and then I looked through the code and I couldn't find any and I knew we had quite a lot. So yeah, it's not Osmo rate counter, it's rate counter. They only increment, though the value you can pass to increment it is a signed integer, so technically you can decrement them but the logic will, well maybe it makes sense but they're only, should only be used to track rates of events which usually just happen monotonously. It automatically tracks the rates of these events on a per second, per minute, per hour and daily basis. In that case it's not really a rolling average but at the beginning of each hour the hour count is updated and at the, or not the beginning of each hour but like one hour after the start of the program. So in practice it shouldn't matter because the updates before roll over into the into the next higher average or aggregation I should say and with the rate counters, with the Osmo rate counters, rate counters, we also have the concept of groupable counters so you have a rate Osmo rate counter group for example to track everything that a BSC is doing from the view of an MSC and can also have an like MSC group to track stuff that the MSC is doing and keeping track of and within the groupable counters you also have you can have independent counter groups per peer so the MSC can track for each BSC separately how many times it has reconnected for example and so per peer and tracking per subscribers also possible and the rate counters are most widely used in the GSM projects they're about well an infinite number if you count all the individual counters that you have for each subscriber but think like 30 through 50 different items yeah Alex? When you say it is possible you mean it's already implemented or it's possible as in like someone could sometime implement this in this way? What was what did I say was possible? Yeah it was a per subscriber counters per BSC counters. Yes that's already implemented and so the functionality is there and also we have for GPRS we have or the GB proxy has per NSVC rate counters okay thank you so yeah this is how you use it it's a bit more complicated you need to so I'll start with the second part first you have to define a counter group just so the first is the name the second is the description so you can basically yeah know what it is then the stats class global means it's a global thing you can have per peer and per subscriber classes as well then after that you say how many counters you have in your counter group and finally a pointer to the actual counters or the description of the actual counters and then the topstruct the rate counter description for the rate counters is just an array indexed with with an offset that also again has a short name and a description what it does and afterwards you just allocate the group and when okay that's cut off here and yeah afterwards you only or in the most simple case you only incremented by one and you reference the actual counter there's also a function as I mentioned to increment an arbitrary amount even a negative amount but in the in most cases you just want that um yes okay um I just noticed you have a a dot in the slider handover dot completed if I remember correctly we use the dot in the control interface to separate is it yes I'm coming to that and I'm pretty sure that we'll we'll talk about that as well in his um ccc review but well spotted sorry just one more question is it also possible to notify uh some some external entity about each event when you call a rate control increase to notify right when this event um fires uh not from the rate counter or any counter system but I mean it's possible to dispatch a signal which is what is used in the code in in cases where we need that so um because then in in the most simple case this will just increment um memory somewhere or value in memory somewhere so we uh I I'll get to the reporting where we have a timer and then we go through all the counters and report the values um yes the the dot separator let's leave it at that for now um so through the vty we can show statistics which um for the example we have the handover completed which is zero events per second and per minute which the reason for that is that it's already quite some time ago and so we have one per day one per hour at the moment but less than one per minute um and yeah the next one is just uh the peer based statistics which have the the number of the peer which is of course then an individual designation uh and uh statistics about this peer so um the newest counter implementation from 2015 is uh osmo stat item um it's in concept similar to the rate counter um we also have groups we have the global peer classes but it's not a rate counter um it just stores arbitrary values and um keeps track of a bunch of them um the items or the stat items can have units which is quite nice and I think um something we should implement in the rate counters as well because as you saw there the name actually has packets at an s level and bytes at an s level and um having basically p or pkt and just uh bytes as unit might be nicer and and yeah tracking that and automating that um and yes uh it'll keep track of multiple measurements so you have uh I believe this is used or supposed to be used for averaging but uh when I looked at the code that reports the values to to stat city um only the last value is reported back so averaging is not used yet but um can implement some custom reporter or anything and um then do averaging yourself so and also for the osmo stat item I have only found one item implemented within the whole of the the gsm osmo com world which is the nsvc alive response time in milliseconds as units and uh yeah but only one is not very much um you're again you have a a group where you specify the name the description you have um an array of actual stat items uh with again the name the description then you have the unit and the 16 is um how many how many values should be kept and yeah you again allocate the the stat item group and then don't have an increment but you have an osmo stat item set um where you just set the actual value and in vty you just get the response um have a feature matrix if you're you're looking at at the counter so we have uh basically should have been orange uh the dirty gray there um so the osmo counter is in my opinion legacy it's not really deprecated in the code but um I guess we have full replacement with the rate counter and osmo stat items um those two are supported um the osmo counter has some counters 10 the rate counters are are quite um prominently uh present in the code and the osmo stat item um so yeah is just one so probably was uh I don't know how how it happened but maybe implemented for just this this one item and then was good enough maybe the stats item branch um all of those can be displayed in the vty as I showed um access from the control interface is restricted to the osmo counter and rate counter which I mean there's no reason why it should be just that um it wasn't implemented yet but if you only have one counter then you probably don't miss that um all of those can be exported to the through the stats reporter via stats d come to that in a second and um yeah the osmo counters are just global counter names uh while the rate counters and osmo stats items are groupable and have these per peer subscriber values yes uh when you say that osmo counter is legacy uh you think it should be replaced with rate counter right um most of the osmo counters that are still used are used as rate counters but as I mentioned um so with the osmo counter basically lets you do both because you have increment and decrement so um what is still implemented as osmo counter is basically the currently active calls which is not a rate counter a rate counter would be mo calls attempted mo calls completely completed mt calls attempted mt calls completed the and so that would need to be migrated to a stat item but uh basically the the two new counters um can do everything that the rate counter does and then is it possible to get the absolute value of rate counters and stat items as well that's the first one is the absolute value and then you have one per second uh so yeah and this basically rest the right and that's that's intentional and that that's not likely to change anytime yes that's intentional and yeah yeah and the and I mean for the stat items you always get the I get access to all the values that are you get access to all in the code you do um but through stats d it only reports the last value back cool okay so yes um then with an osmo com or the bosmo core we have the concept of a stats or statistics reporter um and we can so any kind of counter as I said is supported by that um we have an implementation that basically can connect to a stats d instance or write in that format um and we also have an implementation that basically writes to our logging framework so whatever is configured as logging target we use the dl stats um category and log with log level info so um basically there every minute or whatever you you want the interval to be you can output that into a log file or syslog or something is it is it possible to configure interval pure like a group so I may have I may want some stats you know every like five seconds the short answer is no as you can see so the config snippet the relevant parts the stats interval so logging interval is globally configurable only so you can't even change between the stats d and the the log output it's just one global osmo timer that that then um report okay anyway so it's uh just one timer that then calls all the stats reporters and um run basically runs whatever they do and then you know for for both these back ends you have um the stats reporter stats d where you can basically say how much you want to report so do you want only want the the global values or do you want the peer and um subscriber values uh output as well because those can be quite verbose if you have like 20 50 100 um bsc's um and where to log to and um yeah for the for the um log output as well so this um stats interval you said the stats reporters they get called at this interval what if um so what about external polling could uh could I have a stat reporter that queries all the counters and and so on internally instead of I mean you you can have that but uh then you would just uh I'd have to think about it but um generally you probably wouldn't do anything in the in the internal timer callback but uh or in the internal timer callback you could update some things and then um have your own socket and listen to it I think if you wanted to do polling you probably just use the control interface to poll those values that you want to call um and not use an internal reporter because if you want to read some counters you can always do that over the control interface from an external process at whatever time you want for whatever counters you want but just an idea yes so just just one notice that the reason why I'm asking for for for different rates is because some counters for example number of currently um associated sorry currently used channels we right now are polling very frequently because we then do min max in the processing if I can interrupt um you could do that with the stat items because basically whenever the value changes you or with an internal timer you just add this value very quickly once a second or something and then you configure it to hold 60 values in the stat item and then you can access basically the whole last minute um what the value was and I mean yeah the current code just outputs the last uh the last value which might not make sense in that case but you could do an average or you could also output the the average the mean the min and the max and the the standard deviation or something like that so possible but um not implemented right now okay okay so um just uh last slide going forward um I guess it makes sense to migrate from osmo counter to osmo rate counter which it should be called then um or to the osmostat item of course we always need more counters can never have enough um then have some maybe more helpers or configuration options for the osmo stat item to have averaging uh min max um standard deviation and and those things one big thing that has been touched upon is fix the the issues with the separator so the counters as far as I can see they are still called um or they're still separated with a period but then there's some senate sanitation script or sanitation function that replaces the periods with a column to basically make them accessible via the control interface unfortunately the stats d um export format uses the column as a separator for for one thing so this breaks um there um maybe we just want to have different um functions depending on where we're outputting so we so we can sanitize whatever we have as an input because otherwise yeah or we restrict the names even further but yeah I have to check that and then yeah also make everything configurable um and then yeah also add units rate counter because I think that's uh quite useful okay so thank you if there are no more questions I have one simple question why did you choose this image ah okay right so well this is a counter it's a mobile counter and uh so so I actually I didn't google but I duck duck goat for uh for counters and the first images were those and I thought yeah makes sense uh and this is the the old counter it's rock solid also but yeah from open source