 Hello everyone, my name is Aadash Krishnam, I'm working as an engineer with NMOMI for the last five years. And my job is regarding all my journey of five years with NMOMI and four years with NMOMI. So we face a lot of difficulties because like the approaches, the day-to-day approaches is also going around. And the company is talking, started in 2007 and talks about it very, very, very differently. So I am going to explain to you a few of the issues that we face in production and how critical, how important they are for our businesses. So if you think about NMOMI and its NMOMI, so we are basically mobilizing the company. If we have a strategic gap between a publisher and a producer, we serve ads on all the mobile handsets. You might use smart phones and you'll see that we are the guys who are putting ads to the site for that. So we have a, we have around one, many, many users in the network, which means the device IDs of all the users. We want to improve our ad experience to all the users to serve the native ads to them so that it will help those users also for more easier access of the ads for the native products. So we have around 140,000 peak ad requests per second and 26 TV of user data in our network. So when I talk about this user data, it will use all the things like how user reliance, how they are, they are more refined on the main things kind of stuff. So I am working as an open source TV professional in the last four and five years. I have expertise in my SQL also, of course, as well, but mainly we are a position shop. So because it also, it helps with scaling us with our business design. It's not a thing that we have faced these issues and we have stuck on it. The whole disability is also very excellent to show all the native things, all the ideas within Oscar. And we have around, I think, two new external media data stores as well. One is the IRO strike we are using for real time and anti-productive purposes on these things. And personally, I think, because I am from India and in most Indians, I think, because 80 to 90% of Indians are figured, we have a precarious nation, so I am I. So to talk about the scale of post-descript in movie, it's around these kind of stuff. So since post-descript is a art, it's a relation database management system. We have two systems in post-descript, one is OLTP and one is OLAP. OLAP database size is around 200 to 300 TV and OLAP is around 14 databases. So OLAP, when I say, it's also in post-descript. We are not using any proper reservation for the four generations. We have some hardware, better hardware for these reporting systems. And we are using these also for these purposes. So can you talk about all these things like PG budget reports we are using for some analytic purposes? These stats are from PG budget reports. Like, see, 134 million TVs are sitting daily in our production master database server. And we are average of around 80 to 22k per second with a peak of 58k per second. So it includes how much users are actually in that domain, in that area. So when we talk about the ads serving, the most important aspect for our ads serving is latency. So if any latency is quite a bit, we are facing some issues that we are managing, particularly when it comes to our revenues. So these are two stats, like 5 minute average duration is around less than 820 seconds. So because from an average side, we have some timeout setup, where if in this time interval, you are not able to serve ads, then you will have timeout. So this is that important for us when using it. So thanks a lot for our revenues. So can you talk about PG budget? So the problem we are facing with PG budget is around, if you have a small application where you are less than these are running around 10 to 20k per year, but if you have a scale of around 120 to 200 GB of logs per day on one master, like how do we pass this scale of logs in our production master database server? So it takes 11 to 12 hours for generating this PG budget report for one master database server. So you can say that if I need to get some analytics for my one master database server, I need to wait for this much hours. So in 10 to 12 hours, I need to wait to generate these reports. So that's the problem we are facing. Yes, please. Why do you have so much of those? Is it because of a huge number of collections? It's a big login and a pulse connection. We have everything, like which client or which queries are running, which post, which application name, all this stuff. I understand why we are pushing the application name also in the logs. So you need some of that. Yes, yes, we need some things going along. So there is one in itself. So it's all the same, it's a separate disk. It's all the same and it's difficult. So, I spoke about the old APNOL app. So, we have around 4 database centers, we have distributed in 4G local areas, and ad serving works like this way. We have a major fund data center, which is a master database data center, where different applications are writing to different database servers. Like this is master 1, this is master 2, and this is master 4, and this is master 3. And we are using slowly the application to replicate selected tables to one ad serving master because our main relevant critical database server is ad serving. And we cannot write all applications into one database server. So, it's a legacy problem also, like we are also solving the problem. Like don't enable different applications to shape one database server. And enable your application user to develop some ideas to give access to new data. Suppose if A wants to application B wants to, application B wants to use that data application A, it should not be a DB user to data application B. So, it will increase the complexity of the system also. So, push your application users to develop some smaller ideas to give access to the data. So, like in this way, we have around this, this is the diagram that explains the 3-4 master database services. Slowly the application is enabled around this database master servers and it is done into an ad serving master. And since it is a LAN, that's why we are using the slowly the application. If you talk about like 2-3 years back, we were using slowly the application LAN and LAN port, like this way. This is a LAN, via LAN, this is a full geography of the database centers. We have add certain master data that is taken into 4K5, geography is distributed to the database servers. So, the problem is, we are ready to enable 8.2, thus turning the application is not a master, that's why we are using slowly the application. But we have seen a lot of improvements in turning the application into version 3.2. That's why we have enabled sr setup. So, it makes your life easy. So, if you talk about changes, at half changing during the slowly, it's not a easy job. You need to do a verification also, like all sets, all different ideas, and make sure that slowly steps are correct. So, if any mistake, any wrong command, and because slowly is a sequential execution of one step, so it will cause the application to be recorded. If you have any mistake in your system commands. So, if sr data is unseamless, like once you set up an extremely application, there is no problem. The only problem is if someone who has dropped out of the machine master data server, then you are ready to use the problem. So, that can be taken care by a delay on staining the machine setup. So, this is like this, we have this one master database server, and all data centers, we have around 6 to 10 states. And whenever we see any spikes in our life, so I think whenever we see our database data not able to respond properly, we have to build an automatic system using some steps, steps to scale one more instance in the same data center and edit in the reputation. So, that's it. The size is around 100 different quantity. This is the real world point, which is very specific. So, now we are talking about the issues, production issues. So, every time from the last four years, we have been seeing calls, these issues happen, page duty calls, and these are such. So, that's a life of an awful engineer. You have to wake up and see what happens there, what's wrong with the new database servers. So, what are the different issues that you can come to? There are so many issues that can come, so the database is a bottom layer of your whole application. So, anything wrong happening in the database like in network, you are also going to face some issues. So, since the setup is like this, so anywhere you see some network issues, some cities miscommunication, you are spending the rest of your time delaying. In fact, there's no mistake from your side, but you're going to stay some pages, like say your application is really disadvantaged. So, a classified major problem is that you have to go into four sites. What are the user connection problems and application issues? Second is fragmentation problems, the application issues, and some mixture of like smaller issues or whatever. So, you have to create a database server and give it connections to your application user. Now, application users ask, make sure you give like a finite, like a genuine number of users to your application. It's not like say, application users come to you and say like, I need 100 connections. And 100 connections for like, more than 100 connections is like, very, very big demand. So, whether any connections, it is not a simple connection. It asks for many more other things. It asks for them. So, if you, there's clearly mentioned in the PG documentation, we have the number of connections that definitely depends upon the shared memory of your server. And your throughput will also fail. If your resources are not properly allocated. So, and you're going to face this issue too many connections for role heads with an identity transition and all these issues. So, what can we do? What can we do to solve these problems? You can do the client pooling from your application site. It's not only that. So, from the perspective of read scaling, you can do the pooling by using PG function. You should force application users like, see, why can't you do this, this kind of stuff, so that you can do a good application pooling. And you can, you can have some smart things written on your database server where you can have either hang-up or connections. And one more, since we have a geographical distributed database service, make sure your application servers also state some of the same code. Because whenever you make a, like, different code of all, it will also act with penguins departments. So, if you're acting in a different column and database in a different column, it is going to be a slow-run your database server. And yet each component of application is that particular. It's not, like, so that whenever you face any issues, you will, you will, like, check the log and you can easily see that this user is coming from this simulation and from this client. So that you can reach out to those persons and say, why, but we're doing this much things in the database server. So, and you can do it, like code before. You can set up a script, smart script, we can check our time minutes and 15 minutes, like, if there is a state, connections and all these things. So, you can kill automatically. So these are the second parts, fragmentation problems. So, it's, since it's a distribution database system, the fragmentation is, like, we, we have one page application which is doing around in one minute around 40 to 50 updates in the single table because we are studying as accordingly and you update our revenues accordingly. So the table size is less than 1, if you check in this output, so you will see that the double count is 30k, but the dead double count is 1 million, 120k, right? And the dead double percentage is 90%. So whenever you space these kind of problems, so in fact we didn't get any fragmentation rate. So we got around slowly today. Like Kevin has also told, if your updates are taking too much time on this fragmentation table, so if this table is fragmented, it makes your updates going like this much very long time. And you definitely will fit that initial because your selection is not going to be more or slower than with this fragmentation table. And if you check the original size, 30k tables with the couple lines of this is around 1 million, 30k. But at that time, when the dead double percentage is 90%, the size is around 3 to 4g. So it also consumes more days on the database server. So that's why how many solve these kind of fragmentation problems. So of course this could also provide some things, but these things, the first four things, you can set the more aggressive you are auto-wetting with your table size algorithm. You should not wait for the default auto-wetting program. You can set auto-wetting parameters according to your table requirements. You know this is a very, very heavily updated table, and you're going to set some parameters of these table only so that the auto-wetting could completely find out all the time. It will not make that table that loaded in the database server. So if you see like these things, like threshold and annual threshold, it directly depends upon the extreme number of rows in your table. If your table is around 200 days, you can set the threshold. But after these new updates of tables, it will just back into the learning algorithm. Or in percentage, where you scale certain, where you analyze certain, even mention percentage others. Apart from this, but if you still see some issues, like your auto-wetting is not able to finish properly in some, like not able to de-flagment the table, then there is one more very good kind of a field factor. But this thing you need to be mentioned when you create a table, not after you see the segmentation issue. If you know this, if you talk to the students, people will see this position that this much table is going to be heavily updated, then you can exceed the table in this percentage, like 10-20%. So what it does, so internally, tables are stored in series. So what it does, it will not fill the page with all the rows. It will fill only 40% of the page and leave all other 80% in blank. So that when any new update comes, the updates will be written only in the same page. So that this way it can help. So it can help you save space also. So next, this is very critical for our application. This application is very critical. So many times we see issues like with servers, with network. So what are the different kinds of network issues you can face when you have this kind of setup in your production? So the first issue says that your standby is going to be out of sync. It's of no use if you face these kinds of issues, like requested wall segment has already been removed. It means the master database server writes wall files in its pdxwork directory. And your streaming application delivery is that much that it will automatically purge the older raw files, which is not yet updated to the server. And then it will not be able to get it. But this functionality is like, if you go 9.4th version of all this stuff, there is a replication slot available which can take care of these issues. And this could not send data to wall stream. So this issue usually comes many times when you have some issues with your NIC, networking process, startup, some network, some ethernet card issues when you serve. So that may be some package dropouts on the server, or some other kind of stuff. And this last, could not open pdxwork. So when you have a master and slave, when you make a slave as a master, so how streaming application works, it creates a sequence of wall files, the name of a wall file. And next time when you have a third slave, you are going to create this master, it is not able to understand the writing sequence of your new slave, new master. The next part is, so one more thing in streaming application is you have to set max connections equally as in the master and boss in the slave. So if you have slave as lower max connections, it will fade to restart. It will say your master is saying that this much max connection is there and I cannot start this less than that. So the last, and when you have this PG based backup, about in backup, you can see the issues. When your master database server might, because every time you set up a slave, you set up an entry in your master database server that this slave is going to connect to this master server. You should have this entry in the DHBCon saying that it can cast this slave user to connect to copy that up in the master server. So if you fail to miss the entry in the Ph1, you can create this PG based backup recording by the manner. But this repetition setup, so few things which are very important is you should keep an eye after the number of clicks of file generating area. We have a script which will have these number of posts. So we have some network locations like from different, different core database centers. So all of a sudden like Friday you are okay. One day you are looking to say that your wall finds an increased drastically. So if you bring up 300, 400 wall ties per hour, so one day you are doing it like three to four 10 x increase, then definitely you are going to face some network issues. So if you have some like clear understanding of your network bandwidth also, like how much data can be activated from this master to this slave, this data center to this data center. So like I told you before, like PG that is also implemented so that you will not face out of sync slave issues so that you won't have to reset a new slave. So one of the good things that recently we have faced one issue where this, so this is a like production setup. So we recently faced one issue where business team wants a new replication kind of a small polo setup in new data which is different distributed to another data, another place. It's not in these four data centers. So they need this like in a very short time. So but network team is not able to like say if it can't be possible and need some extra like extra deal with other network provider all this stuff. So, but we set up a like product internet channel, the new setup. So we set up a new data center on the internet and say let's try this, but the amount of whole page generation is that much that our new data center is not able to catch up properly. So there is always a delay of some likewise because the network is not that great. So what we can do with assets currently in this it can compress your log files which is going to be replicated to this data center. So I won't see any improvements in like some compression of these log files in the internet as well as maybe it comes in future. So if you want to reduce your data application, the amount of data replicated, you can do this assets running which can like compress your number of log files and so that less amount of data you only need to be able to get to your new files. From your experience, what was that comparable to the previous one? So it's just basic compression of assets which assets are a simple command of assets I haven't seen. So you can set a tunnel, there will be new tunnel between your master and slave like on a different port. So what it does is it will treat as a network layer on this master and slave. So every time when these x log files complete copying it uses that compression data. Not this... From your experience, what was the cooperation level? Like it would reduce the chart by 50% 10% or... So we have... I don't know the number yet, how much percent it is but we have seen delays of well last 14 hours. This much delay. And we are generating around 300 to 400 MB file in this XM monster. So 308 to 16. 1 file is 16 MB. So it's around 3GB of data. So... 16 into 16 to 300. Approximately 3GB. So 3GB of data So then it is struggling. But after enabling this as a compression you haven't failed this. Even in our delays, it was something that was that reaches 1000 levels. So after this enabling we haven't failed any issues like this. After this assessment. There are some other issues also. Like when you have around your reporting updates where we have around 14 data files. So we have around 8 to 10 SSDs, on master server. And we have created table spaces like we currently on these master servers. 10 table spaces. So if any new requirement comes and you want to create a new table space. You need to make sure that this directory and this SSD also created in all your applications. It's not like that. You are adding currently the master and you haven't added it in the space. This is not an exception. This cannot be added. Reduction is broken. Make sure that you want to create a new table space on new SSD in master server. So next are the two issues that could not be brought up. This is the operation. So there might be some issues where you can see the data files issues. Your indexes are not going totally above. It's not even read properly. So you identified that table but you are not able to see. I will solve the problem. So long range you can do a backup or you can do a drop-down indexes. See this indexes are causing problems. Or read it to the indexes again. So that way you can fix this. But you need to do some monitoring also. This table is stagnant. You need to focus on these errors by using Klan or some other stuff. So Klan will generate an alert of any kind of errors or panics that come in your OS. So when you talk about monitoring and alerting, few things which are very important to monitor. Your connection stats, how much connection is ideal around this, how much is waiting on all the stuff. And since some of the everything is in Klan, we have some alerting on Sony also. So that table stats are also important. We have done with PGStats how much table is fragmented, how much table is, how much table like indexes, how much indexes are properly used or not. And transaction stats, how much begins and commits are happening in your database server. And this is a great update and it's not an option. We can get these operations by your PgMeter report also. But doing a real-time and alerting is a challenge. It's not easy to do a real-time operation like this. And Klan system stats, Klan traffic is long, how many connections are connecting and how much they are using those connections and all these stuffs. So that can also tell you about this, all these tables, views and functions which the server provides and how can we get this information. Like you can see all that are equally between PGStats and PGStats and on the PgStats there is PgStats relations. So PgStats all tables give you all views of how is your table, how much sequential access, how much index index access is happening on this table. And these functions are also like you can see what is happening with applications. The PgStats is around which patterns of your tables like how much tables is implemented, how much couple count is there. So that's a very useful function to get tables and information. Any questions? So that's what we're discussing, like how can that, doing a real-time monitoring and I think it's a real chance. How can we, so if you use any like property solution like Kevin's always taking new rally monitoring and do it, but doing a real-time monitoring and I think it's a tough thing to be using any of those tools. So if you if you do a, like if you ask anyone like how much the gentleman do a monitoring and I think I will say nine years and checking and I will say, everyone is doing like today, like you run a check on your plugins and do a learning on like for many simple db2d, like this is a class 3 from PgStats and getting into this and learning that but we what we are doing differently here is we develop this system like how can we do a real-time learning by using a mixture of diamond, darsana and graphite and logStats. So this is like only one, one diagram which can explain all of the setup. So we will have some expertise of Python so that we wrote some Python custom scripts and get as much information from the database permanently and sending it to your graphite servers. So diamond is the collector which is collecting this much information and it is running every minute on all of the servers so every minute it runs and gathers all the needed information from all the servers and send it to graphite and after the graphite since after the graphite we want some like some dashboards to look under them some dashboards at level, add dashboards at database level so we have to get an amount of graphite for a better graphite and when it comes to the alerting part we are using this matrix is only for alerting also. So we are sending this matrix to graphite and there is a check effect plug-in which we are using and which reads this value and allow it on the basis of this. There are a few things that we can you can say like best practices or 10 commandments which we can try it out like first of all like capture admin status as you can since we also provide a lot of functions views and classes you can use different functions to read all of that and and CC it is not necessarily you can alert on all of that it is very important you can have all of that all of that because sometimes you might to create some problem what is happening on a database service these much other stats you can capture it will definitely have to be well done so one more thing like as everyone said test your backup calls so it is not backups are not important we store is also important so we are sending FDR we are doing backups so we are sending it to SC and we have one script which also and whenever whenever some issue comes make sure you have your work on a copy if any issue comes make sure you do not work late with your main critical database it is pgx log, pgc log you can play with pg log because it is not it will not have any backup but make sure you will not touch this pgx log which connects on your wallet and on all of that and whenever this app saying full page writes it will help you all whenever there is a reboot or crash on an active database server so it will make sure that the active data is not touched so every time it is doing full sync pricing on each array of the updates so when you have different kind of applications running on your database servers make sure your updates are the same but it is not like you are missing many too many applications with one database server and giving access to all the unnecessary people and you one more important thing is you evidently have a delay but it looks only on your database matters say this much transition this much dismay it will definitely help you if you know your application also and if you are able to relate an application problem with your database problem then definitely you have a move on on your database so understand your applications also what they are doing and how much significance of applications are there and how they are communicating to your databases and it will also help so since alerting is like this amount of database servers production database servers what we are doing now is we are grouping the alerts so since we have some timeline based matrices on the graphite we are grouping alerts on all the color level we are not doing alerting on individual core servers imagine you have 200 servers and everything goes wrong or something you will get some little alerts on that color it is better to group the alerts and check the value and alert on that if you make it a live page you don't need to page or you don't need to get alerted 20 times or less so few years back we were facing a lot of issues so what we did in our daily schedule is we had to really stand up and understand the significance of all the alerts whatever is coming so we spoke all the things why it is coming if any application in southern south you need to call those people also all that each and every alert so that the alerts will reduce consistency so daily we are around 8 to 10 maximum alerts so if something is wrong it also depends on other matrices on the network also but it is very less compared to 100% and it is better to alert if there is something wrong with your database server you don't need to wait for some application to come and say your daily is low so whether you should get alerted first then alerted to the application page people that are calling you then you have to get a database server so are you using streaming replication with async slaves and then you are cascading down the multiple levels async is all so it is not simple async is simple but streaming replication and then you cascade to the multiple levels ok and then what are you guys using if you are really low balancing yes so there are two types of network that we are using one is we are using a network layer virtual IP we have created a virtual IP in that whole level we are using a big virtual IP and it added all the IPs of all those IPs so every time newslays are added and we found the network themes in this IP so this is for one application one more like smaller you can do it at Jetbox ok thank you you have to do a project teaching streaming for hours yes that's a good question so I didn't know about that part so we were using LogStash to analyze all the labs and we were sending methods to elasticsearch and from elasticsearch on top of it we are writing departmental we can also write so we have that matrix but we can't control on that expertise in that we were able to just how much leaves and these much plastic is coming but some more intelligence need to be based so that's our first value item