 So, I am Abhishek, I work at Helpship, I do the DevOps here, I will be sharing our learnings with MongoDB. We have been using it for last 3 years and currently we are serving around 10,000 queries per second in our current MongoDB infrastructure. So, we started with... Am I audible now? So, how many of you are using MongoDB? So, how much people? Let me introduce MongoDB, a little introduction. It's an open source document database that provides high performance, high availability and ultimate scale. So, this is basically a MongoDB structure, it looks like. And it looks like JSON, basically a JSON documentation. When we say it's schema-less, it's called schema-less. We say that Mongo doesn't require you to have particular schema and you can have any specific format or data. So, this is basically a simple longer document which has key value pairs JSON-like format. Many of you must be familiar with JSON. So, this is why we chose MongoDB. Basically, it's schema-less as I already told you. And then, faster development and early stage. So, MongoDB is not complex like other no-scale solutions, for example, React. So, it's very straightforward and simple. You can just install a single MongoDB instance on your local system. And it gets started within 10 minutes. So, that's what the faster development is at this stage. For the start-up, it's very much important to deliver the product faster and get it devoid or get it evaluated by the customers. And then, we first started with master-slave replication. And then, replica sets, the current, important, or hot feature of MongoDB was not available. So, we started with master-slave replication. So, advantage of master-slave replication is it's easier to maintain. But the restart advantage is no automatic failure. So, suppose if your primary goes down, then there is no way for MongoDB to automatically tell the secondary to get shifted to primary or get promoted as a primary. You will have to do it manually. So, we have some of these S&T operations, which are very common actually. Like converting master-slave to replica-slave, converting slave to master-slave. Those kind of operations require very, I want to say huge, but it requires a downtime basically. And again, there is no automatic failure. So, you will have to do it manually. So, that's what the problem with master-slave replication. Then we have global locks. This is another debatable topic of MongoDB here. I want to tell you a lot. And back in the point, the version is MongoDB used to take global lock on entire database. So, for every line application, there is a global lock on entire database. And other operations are possible. Suppose you have two databases, two databases stored in the MongoDB, and you are writing something to one database, then you won't be able to write to another database because there is a global lock. That was the problem with 2.2 version of MongoDB. With 2.4, they have come up with the database in their locking, so there are no global locks in it. So, if you are writing to one database, and there are other databases, other databases can accept reads and writes while one is just accepting writes because it's locked. So, they solve this problem. And they are coming up with roll-level locking. So, it's even better, but it's not a production-ready idea. Then 2.4 still has global locks on certain operations, but they are very limited like repairing database or compacting database. So, repairing and compacting are very critical operations. That's why they have global locks for only those operations, but not every write operation. So, roll-level locking means you have documentry. Everything is in form of document. It's not the couple of the RDB. Yeah, but you can read the document like a book. You put a lot of documents. Yes. No content. Right. No concurrent documents. So, yeah. Then, these switch to replica sets for... Because the traffic was increasing. Customers are increasing. And it's not really visible to have master set of application in place. It's such a high-request website. So, these switch to replica sets. replica sets give automatic failure. So, basically, you have a primary. And then you can have around six secondaries. You cannot go beyond six because it's not recommended by MongoDB. But you can have around six secondaries. And you can place them in different... We use AWS. So, we place them in different visions for... And we can have availability zones for failure or AWS failure. So, you can have six secondaries and one primary. And it works like that. It takes up elections. And each election chooses primary and secondary. So, if your primary goes down, then secondary will stay down. As a primary. Then you can... It's called RS company, replica set configuration. In replica set configuration, you can define priorities. Something like this is a primary with priority five. And then this is a secondary, priority four. And then we have another secondary with priority three. Suppose primary with priority five goes down. Then out of these two secondaries, the one which has more priority will become primary. That kind of stuff. So, this happens automatically. You can set priorities yourself or... The priorities are assigned by default. So, that's certainly failure. And rights for loss without rights. So, yeah. So, right concern is basically... How many members of data base you want to write? You want your rights to go missing. So, suppose in three notes, replica set cluster. One is primary and two are secondary. So, first type of right concern is unaccommodation. So, right comes from client. And your driver, which is writing to the primary, will not acknowledge anything to the client. That's unaccommodation. It's very fast. It's very fast, but maybe unceasingly. Because there is no acknowledgement. If there is a network failure, your client won't be able to take it. If there is a data loss, your client won't be able to take it. That's unaccommodation. It's very fast. This kind of unaccommodation right concern may serve because of particular type of obligation, but not in our case. Then second is how a case acknowledged. In that case, if there is a network failure, then only your client gets acknowledged. Otherwise, third type is journal acknowledgement. Means when a request comes, it gets returned to the journal and then your client gets acknowledged. Basically, in that case, you are sure that your data is returned to the journal or to the client. And then another type of acknowledgement is there. It's called replica set or replica set. In that case, what happens is your right comes, it gets returned to the journal primary, and then it propagates to secondary and gets returned over there. And then you get the acknowledgement. This is called replica set acknowledgement. So without right concern, we were losing some rights. There was a height. So we encountered that problem and then we started using replica set to make sure that our rights are going to the secondary journal. So priorities, I already talked about priorities. It's used for election in purpose. So primary goes down and secondary goes to the primary in that case. So you might want to set a low priority to one of the members, which is in another region, and your main infrastructure is in some different. So you can set up priority on that secondary service so that it will become primary. And then promotions and state accounts. Promotions and state accounts, you can... Mongo provides their own shape where you can use RS config to promote and devote primaries and secondary on the fly. So you can just write a single command where you set the priorities. So this is the Mongo now. Let's say it goes from here. And then we have priority here. So this is configuration. So suppose for secondary, we have to... one of the secondary members, we have written this configuration. And then you just do RS not read complete. So what can happen is this way you will be able to change the priority of earlier Mongo users to something 2 or you can set it to 3 or whatever. That is how you can change priorities on the fly. It has... using this RS config to promote any primaries or devote any primaries to secondary, or promote any secondary to primary. And in that case, what I must call a fraction of seconds. All your Mongo's are secondary. So no writes are going to the rating asset. In that case, you can use some writes. Again, we can address that problem through your application or viewing system. It depends upon what your application needs to have. So that problem can solve any problem. Then... So you can... No, it will be this fraction of seconds. One to two seconds. But then if you have a high traffic, then you will lose the writes. So, yeah, sharding. We pride sharding, actually. We are not using sharding in production. But we pride for the generative purpose. And we... We face this outrageous problem, actually. Because if you are shardy, if not chosen properly, then it happens that if you are running, let's say, four instances, and your one connection or in other words, one table is distributed across four instances, okay? And you are writing data with that table based on your sharding. And if your sharding is not right, then it might happen that your data is getting data single server only, out of those... Your table is distributed across four instances. But your data is continuously getting written to only one server. It makes it a hard way, and it might go down. So, choosing the right sharding is very important. They have extensive documentation about changing sharding. One of the early developers from Korea have written a book about it. It's a good read, actually. Then, yeah, the sharding... Taking backups in sharder environments are also... I can say, creepy. Because... Simple monger dump... They have this command for monger dump. You can just grab all the data in a single file. But it's not possible with sharder environment because your tables are distributed across different instances. So, basically, when you take a backup, then you should be able to restore it and use it as if it's a new data. So, that part is actually tweaking with sharding. But you can use file system snapshots and monger dump as well in case of taking backups. But it's tricky. This is a very interesting feature of the read preferences of nobody in the previous analysis. So, by default, all the read operations are written from the primary. So, even if you have secondary, secondary is just to make the data available and in case of failure. But if you have a default mongo installation, then all the reads and writes are going and coming from primary. So, we face this problem that our application actually... Actually, there was a read queue. The read queue was found and one of the promo code ran. And what happened is there was a lock on the database layer. And basically, since there was a lock because the write was going, the read queue found and some of our reads started to fail. So, that's where we discovered that all our reads are being served from primary and secondary are lying there, just like that. So, in that case, what we... Pardon me. So, in that case, what we did is use secondary preferred read prefer. So, what secondary prefer does is it allows your application to read from secondary. So, in that way, it reduces the reads you found on primary and you are able to serve writes through secondary. So, basically, read preferences are used to scale your read operations and our infrastructure, our application is very read-tennis-happy. So, we had to use this feature to get all the problems we were facing. It's very much possible that there are propagation delays in this scenario where your application is writing to primary and your application is immediately trying to read the same data which is written to primary from one of the secondary because we have enabled the secondary paper. So, your application is writing to primary and trying to read the same data from one of the secondary. But what happens is there are a lot of requests. It might happen that your data is not propagated to secondary. So, in that case, it will fail. So, that is very much possible. Secondary might have a scale data. So, what is the propagation time? Same time. Propulsion time is... we have observed not that much as long as you have all the secondary parameters in the same area. But if region changes then there is some kind of delay. I don't have to remember right now. But there is a delay. Then, yeah. How would you be... you can even check the consistency states of the data. It might happen that there are propagation delays. I have already told you about the right consistency, right? So, yeah. So, one thing you can do you can combine this in reference feature and write concern to make sure that writes are propagated to secondary. So, you do that. Your application might slow down a bit but your data will be consistent. So, that the problem won't happen like your application is trying to read the data from secondary and it doesn't have a data. It can be limited using write concern so that you can tell your application where till the write goes to every secondary at the primary. Yeah. So, it references it reference actually helped us in one more way that there was a failure of one of our primary servers some AWS and then we had to promote secondary to primary and as I have already told you that there is there is some little time fraction of time where you are all normally instances are secondary. In that case all these are writes are failed reads are happening with writes are failed. So, having read reference said we could make sure that primary is not getting all the writes and we could save some data because writes and reads are distributed across all the secondary. So, we could stop, we could avoid all those where in first case all the writes and reads are coming from primary. So, all the reads which went to secondary was not data also. Yes. There is another feature called tags. Tags can be added to secondary and using tags you can tell your application to read particular data from particular instance particular secondary. Say for example some of you manage to figure out particular request is coming from US and particular request is coming from India and you have one secondary in India and one secondary I am just taking some hypothetical situation. So, one secondary is in India and one secondary is in US and you can tell your application based on these tags to fetch data from the nearest instance. So, US traffic coming from US will fetch data from US instance and traffic coming from India will fetch data from Indian instance. So, line instance for that tags can be used. Tags can be used with write and send as well. So, you can tell your application that this write should go to this instance first and then other instance. That way you can use write concerns and tag division. So, let's come to monitoring. So, MMS is called Mongo monitoring service. It is provided by MongoDB, and it is basically a simple command called DB server status which gives a huge output and lots of insights about how Mongo is performing and all. MMS is basically the same information presented in the command graphs like this. So, you have all these information, lot percentage, queues, cursors, network connections everything you get in the document format in Mongo sharing itself using DB.server status all. This is presented here. This is a free service by the way and I would like one to use every MongoDB user then backups. We are still currently using MongoDB to take backups. We use generally secondary instances to take backups from. It might not be a point in time backup, but it has a purpose so far. But it has its own problem like when you are doing Mongo, the system usage on secondary goes very high or it takes up RAM and CPU as well. What is this concept called working set in Mongo? We are frequently as a data has to fit in the memory. If you don't have enough memory, then Mongo will start swapping and swapping will definitely do your performance. In case of Mongo demo you need a lot of memory as well to take the data is done. It might affect the performance of your Mongo cluster if you are using Mongo demo. We have never used a backup service by MongoDB but what I read about it, it's a good solution to have backups without affecting much your cluster performance. I was training the working set needs to fit in memory. So, frequently accessed data needs to fit in the memory otherwise Mongo starts swapping and it degrades the performance and increases page points because Mongo is trying to access some data from memory and it's not there then it goes to this and checks for the data and then there you start having performance points. The second thing is required database. What happened is earlier we used to store our analytics data in MongoDB and that was huge data we request everything we were just putting into MongoDB so I can let our connection size do very fast later we should use some other technology to address the analytics problem and we stop writing to MongoDB but we have these huge collections of useless collections in MongoDB so we decided one day that we know all those collections and that was around 70% of our total name so we removed all those collections but interestingly we found out that the database size is same it has not changed if you take MongoDB dump then you will get around 5 MB of 600 data and the database size never changed even if we deleted around 70% of our data so when I looked up in MongoDocs I came to know about the compaction and repair database name so the repair database what it does is basically the refragmentation so it checks on the collections and removes unwanted data and refragments which reduces the overall size of the database but this repair database operation is very expensive because it takes a global lock on entire database you really don't want your database to be locked when you are still serving the request and then again it needs twice the twice the disk space to complete the operation so that was the problem with the repair database so we still have the huge database and not much similar data then XFS so all of our servers use XFS it's recommended on MongoDocs basically XFS provides journaling and it's very well supported for databases like MongoDocs so we use XFS and XFS gives a nice feature called XFS Reach where you can use that feature to facilitate file system level snapshots of MongoDocs and then it also provides XFS RoHS which you can use in case you need to expand your disks so just attach one more disk and you can just say XFS RoHS and it will just expand disk and live expansion basically you don't have to get anything coming up then yes since we are hosted on AWS we we try to figure out what is the best solution for storing a database on AWS for Mongo then LVM and RAID 10 is what we use LVM we use for extending the file system so you can attach more disk and expand and RAID 10 is basically for redundancy one plus one plus zero then yes there is one terminal level of system level tweak that we use is keep alive time which is we set it to 300 it is recommended for sharding and recognition purpose that particular keep alive time these are some things so we are we are running to use or we are rather evaluating right now this is another solution called TOQMX TOQMX is nothing but another MongoDB but they have come up with different techniques of storing the data all threatened with so using threatened with they manage to reduce 90% of space then they also have contractions the asset the RDBMS databases which are asset those contractions they are providing which are currently assets in MongoDB then and they also claim that they provide takes faster and less so we are running it out right now they don't provide these two services which Mongo Current in Word takes to text search and especially in index so so how many how many people are involved in beginning this setup currently I am I am I am