 More VoIPa monitoring we got Oleg here who's gonna tell us about CP3. It is his first ever public presentation So please give him some love and encouragement Thank you guys Thank you guys. Thank you for coming I'm going to tell you about yet another voice all right be monitoring and troubleshooting system called CP3 You might be heard about tapir Sorry louder. Oh Okay, you might be heard about tapir tapir is previous version of our system and But I'm pretty sure that for most of you it will be something new today. Now. Let me introduce myself That's all you need to know I'm agafox everywhere in github and in social media But a bit more I have been designing and developing and distributed systems different distributed systems as developer and architect for more than 10 years and last six I'm spending my time in telecom Telecom did lots of great stuff last couple of decades, but it's obvious that you know, we need to update and adopt telecom to modern technologies and to modern Stacks just to make it Up to market. So that's what I'm trying to do all the time in CP3 as well and on my previous On my previous places. So now let me start from a scary story It's called just 3261 There was a company There was a company who built a great pretty good voice over IP provider service and at some point they just they just realized that they have something like 20,000 messages per second and Engineers who designed originally and who built all these systems. They're spending most of their time Just you know trying to help with customer support tickets because Customer support tickets is a simple thing like I I had I tried to call from a number to be number and I Had no success. What was the problem and this call was when when was this call this call was Something like three hours ago, but when exactly I don't know maybe 15 20 minutes difference just imagine now that 20,000 cp messages per second and 20 minutes It's a good amount of data. So this company is a store TCP dumps and they rotated it by time and by size and then They tried to find and correlate CPS session on every I mean all correlated to this all related to this call Information it from these TCP dumps. So it wasn't saying Another thing this company They didn't have much of budget, but they asked us to help we decided to help and the first thing We did of course because we are lazy engineers We tried to make a research. What do we have on market? What do we have in open source? Of course, and let me recall they had a simple requirement Release pull engineers and make support team responsible for troubleshooting. It's easy, right? Just a number be number and time. That's it We did deep deep deep research by deep research. I mean that we looked far then second page of Google Then first actually and that's what we found so far It was Oracle Paladin. It was Homer And it was white monitor. So it's three solutions unfortunately, we didn't have much budget to deploy any of these and We took Homer back then it was Homer 5 and unfortunately 20,000 cp messages per second They couldn't handle it in open source version because it was my school and it was far. It was two years ago That's why The only one thing we could do is we created monster we created tapir and This monster did the job. So tapir was able to deal with any amount of data without like any problems But okay, it was restricted. It was Developed for particular case, right when you have like a number be number and date. That's it So we made poor engineers happy Yes, and they kept going and building new business logic and customer was happy as well Tapir was tapir is based on lots of open source frameworks and libraries and projects for capturing for processing for Storing data, so We couldn't to do not pay back to communities. That's why we pushed tapir on github. I still remember that night when we gathered these guys we We had some beers and we did this get push and we started waiting for glory and growing community Next two years it was two years ago next two years We didn't sleep we didn't eat because we were fixing issues and working with pull request Because it was crazy and now I'm proud to say that we have the most start telecom project ever I Think you can do like this here Okay, that's how we dream reality is a bit different reality is a bit different so After two years we have 36 stars. I think that 35 is from our friends and 36 is from my mom And But the worst part that we had only one open issue and it was like a question and something like this So without marketing you can and in open source nothing without marketing But we truly wanted to make the project good What we didn't give up as you understand that's why I'm here now We just took a break. We kept working on our main projects and main activities and meanwhile we were collecting information what else voice over IP providers like different but mostly like big-size providers want to have and want to let's say see in our type of products and Collecting this information came to simple requirements number two So release poor engineers make support team responsible for travel shooting release poor support team and make computers responsible for monitoring to prevent travel shooting so it's easy What could we do We did see three see three the next version of tapir and now it will be like Original I know it will be the name of brand and product We are going to work and develop. So let me introduce you Let me give you some technical details because I was explaining some stuff so far This is our architecture diagram Pretty common for any Monitoring platform Absolutely, I mean it's a typical thing. You just can change names and that's it. So captain kind of stomach it captures data it It's a doctor so it encapsulates different protocols from I Mean network like row protocols from salta and sense information and internal protocol to salta Salta is a beating heart. It's an engine. It's even driven pipeline Based on okay, all our product made written in Java and Java based Languages for instance now new version of see three written on Kotlin Kotlin is the language from JetBrains guys behind Intel G idea So Salta is pipeline and it's responsible for Retrieving data from different sources. You can see that we can grab data from third-party sources like open sieves we switch hysterics and Salta aggregates correlates all call related information partially correlates not fully and Sends data like metrics to third-party Monitoring to influx DB because we use influx DB and to MongoDB tweak is our brain because we correlate information only partially because you can't correlate it like Fully and you can't correlate it not all not all information comes in real time That's why tweak makes lots of work aggregating In other part of information and hoof is just a beautiful UI That's it. I skip database layer because I want to spend one more slide on it For payload we use MongoDB and we optimize that a lot Mongo distributed and chart by itself So it actually has a very good performance, but to be able to handle Any amount of data we do lots of tricks. I mean kind of best practices first of all for seep We do a couple of levels of partitioning partition on partitioning level one we separate Information by seep methods like by seep calls and seep registers on partitioning level two we Separate data on call index because you don't need actually to index every message you have for instance If you have 100 trying why do you need to index it or if you have at the same time? Why 100 trying has the same fields like from to whatever so that's why we Have index created originally from initial methods with some additional information from another methods and also we have partitioning by time Because without partitioning by time you can't actually build a good monitoring system. So all these optimizations they help us to Have search agent and such a nice feature like advanced search It's a new one because as I said tapir Was able to search only by number like a number been I were number and daytime here If you see advanced search new dashboard where you can search by any type of information you want Here it is. It's a Real example from production you can see that engineer he looked in the beginning. He looked for seep register and State unauthorized he found out that okay. I see some anomalies. He put Collar mask in seep register and he found out that okay, somebody is just doing registrations every five milliseconds Just you know Increasing number and then he just checked that okay all these things comes from the same I Source address. So it's a fraud detection. I mean it's a real case of troubleshooting of fraud detection I just changed numbers of course But to realize that you have some issues that you have some Anomalies on your network, especially when you have 20,000 seep messages per second or some hundred thousands You need metrics you can't live without metrics and you can't have only for instance average code duration or Okay, you can have even average code duration and ISR is metrics But you need to have multiple dimensions you need to have ability to see what are the average code duration or different metrics Let's say for particular customer for your Interconnection partner, whatever because you can have a partial Disruptions on your service and you need to know about this. So that's why seep 3 Correlate collect and correlates metrics by any dimension. So you can say, okay, I want to have it by Trunk I want to have it by user agent. I want to have it by this or that you can have it here and Here is the real example of our customer company telly stocks They are C pass provider Who is working on restcom platform and they have They have installed restcom platform on Amazon in three different regions in us in Japan and in Ireland and You know till the time they asked us to Provide them some metrics some information about their service. They already had Something like environment with requirements. For instance, they had infrastructure as a code project where we sensible they deploy all these services in cloud and The things that they already had data doc as monitoring system So what we did we just provided them part then part of seep 3 Integrated with data doc How did we do you see under seep 3 you can see this logo? It's a Java framework called Micrometer and this is actually adapter for any Like literally any no almost any almost any time series database you want to you you can Think of for instance this micrometer can send metrics to data doc to parameters to influx db to elastic search to New Relic Whatever, whatever you choose because let's say that in our world now Integrations are like a new feature. I mean you need to provide flexibility and the integrations so Now it's a demo time Yeah dangerous demo So the things that As you know, we started from okay, I need to do something to make it visible right? Can anybody help me because I think it's Okay, nice Okay, so We started from github project as you know and it had no It had no success That's why now we want to build our community around demo project. So we have demo seep 3.io. You can log in there Okay, you can log in there and you can actually try it by yourself Okay with artificial data for now, but we are going to implement And put all our new features on the under this project So I think that to try is even better than to see it on github but github version is coming to so you see simple search is something that we Think is good for support teams because it's something that Replies on most of the customers tickets. You have this advanced search Okay Okay, you have this advanced church where you can search by method by anything you want IP address list of hosts It's again the same thing Unfortunately for demo we did only one leg, but we will add more information and you will see pretty much The good picture with correlation of all legs and all methods Currently it works only with seep no RTP and RTCP And also you can check out the technical dashboard. We just put it As an example you have ISR average call duration call attempts by different states Average call duration by direction whatever so it's just a showcases We will add more showcases and also we will add business dashboard, which will show you like different business metrics because When we realize that okay, we have all informations about network exchange we can use it for business as well We can show you some business insights and this is great Now let me try to get back you see some code Here is our roadmap We are going to release github version soon after that we will be working on DPDK capturing because now we are working on RTP and RTCP and When we talk about 20,000 ship messages per second, it's insane amount of media and Lip pickup actually we use originally wrappers over lip pickup is not capable to deal with this thing. That's why we need to Go on the low level and use DPDK DPDK is toolkit for capturing information directly from Network card I mean from network card buffer without Routing to kernel so after that we are going to implement machine learning we have metrics We have different dimensions and we can predict anomalies and actually even now within flux DB and different Plugins for it. You can have something like time series Metric analysis and everything and we are going to introduce more of UI improvements That's it for I think next may be three four months And actually that's it at all. Thank you. Thank you for Any questions I can share this This is too short for question Actually, I have stickers also if you want because I It's my first presentation as a speaker, but I love I love Conferences and I am a sticker addicted person So I have a proprietary system and we want to make some monitoring and what what kind of Exports do you support? I mean What should I export to the system in order to have these stats? Can you repeat the question Like Okay First of all, we are capturing from network, but if you have a proprietary protocol Okay, we can make an integration because we have our internal protocol and in addition to this We okay, we can't do not take advantage of open source and we can use For seep at the moment we can you can send hip with these three. Yep So in future maybe it will be for RTP and RTCP as well because guys did a great job integrating with other platforms, so We can't we can't do not Take this advantage Okay, so somehow related until we get there Is there any like a docker image or virtual machine that someone can just take and run and Sure in a month and as I said, we will release an open source version and it all will be with docker and also We are working on uncivil part where you can just deploy components Whatever you need. I mean because this system Has different components and you can deploy it with uncivil scripts or puppet chief, whatever you need Data ingestion is based on pick up files Packet captures right now. It's a live capturing or pick up files. Yes. What about encrypted traffic? You can't decrypt it. Okay, so that's the use case and enterprises. It's a good question But we have it only in our roadmap at the moment Because other other data sources Use a chest Okay, what are they? The things that we are a small team and we are working on thank you for a question. We are working on features based on our customers requests and Okay, yes, when we will be big and fast-growing we will implement SSL or if our customer will ask it we will do it for him, right and also If we talk about different connections, there are a plenty of choices because for instance, we were thinking about integration with SS7 protocols because some of our customers are close to mobile operators and we are good in Camel and in diameter, so If they will ask us we will implement this as well at the moment we have road map, which I just showed you and We are trying to move very fast and just to have new and new requirements from our customers. It's just like this. I Have I Have one more question You're saying that you are using have three Is there on your road map to also support other kind of Messages like besides seep like logs and stuff like that And RTCP. I mean you said are you are going to support RTCP? Yes, we are going to support RTP more because it's interesting, but you have elastic search Why do you need us? I mean We are more like, you know experts in terms of Real-time communications in terms of voice over IP. So the thing that all these metrics you need to know how to What actually metrics you need to how to how to aggregate it what to extract and where to put and we want to be experts in this case We don't want to You know compete with elastic search. It's not I mean So you still can use elastic search for your logs and you can use this thing for voice over IP Thinking of attaching a context with a call that's that's why