 Hello, yeah. So hi, I'm a developer working with ThoughtWorks. So today I'll be actually talking about some learnings I personally had when working on a project for one of our clients. And the requirement was we had to scale around 2,000 writes per second. So we were working for a leading lifestyle channel in the UK, Channel 4. So it's a lifestyle channel that had a lot of cookery shows, lifestyle shows, travel shows. And every show had their own particular website. So they wanted to have one. They wanted to build a service, a site wherein you could go in one place, browse your artist, and basically collect bookmarks, build your own scrapbook, collect recipes, and everything. In past, they've had some problem with their existing sites. So it's a famous channel. There are a lot of viewers. And whenever a celebrity used to come onto the TV channel and used to announce that, please visit so and so site and collect this recipe, the load used to actually increase dramatically. It depended on the popularity of the artist. And obviously, the response time. So it was not a good user experience for them. And it was actually hampering their reputation. So the challenge for us was that we had to build whole this website from scratch and have sub three-second response time under peak load of around 2,000 writes per second. So even before entering, our client was the architect. The client side was really excited about MongoDB. So this is about a year and a half back when Mongo was the new kid in the block and everyone was talking about it. And it was boasting about horizontal scaling. So out of the box horizontal scaling. So it told about failure. It embraces failure. So it can automatically handle failovers. Even if your primary database dies down, it'll do a reelection and those sort of things. So we thought probably we'll try. And we had good support from Tengen. Tengen is the company who's built Mongo. We were in touch with them. And we were working in parallel with them. So we started working on, so this is what our initial architecture looked like. Very simple, some front end apps, which were basically serving the UI, some back end restful services, one Mongo replica set. Mongo is the router which you connect to. And you have a replica wherein you have one primary and two secondary machines, which second is basically replicate asynchronously. So we thought probably we'll build some functionality and let us run our first functional test. So we were partnering with another company. In Technica, we were running using some of their tools to run functional test there. So we basically simulated 9,000 virtual users. But we got service unavailable for around 7,350. So it was not at all encouraging for us. We could only scale up to only 75 requests per second. And that we were really unsure of because we were getting a lot of service unavailable at that point in time. What went wrong? So front end servers started actually maxing out on CPU. So we were doing some image processing in real time. So we were scraping images from third party sites, passing their XML. So it was a bit CPU intensive. And the kind of load we were putting on to the front end servers, it was not able to cope up with them. Back end, there was only one Mongo shot, so one replica said all of the data was in one machine. So most of the time, we noticed that onto our back ends, they were mostly waiting on the IOs. So what was happening is when we drilled down into what exactly is happening after our performance run, or you were monitoring our machines continuously during our performance run, we were saying there's a lot of IO wait on back ends. They were actually waiting for response from Mongo and on Mongo machines that they were actually waiting for the request to complete. So I think that that's the most talked about topic in MongoDB. So MongoDB, in terms of handling concurrency, it has a global lock. So it used to handle concurrency very naively, which means that prior to Mongo 2.2, it basically used to block the whole MongoDB process, which is running onto your machine whenever a write used to come onto the machine. So at any moment in time, you can only have one write. And even you can't even read at that moment in time. So every other request has to wait before that write comes. Basically, it gets the response back. After Mongo 2.2, they've removed the granularity or reduce the granularity to a database level, which means that instead of locking the whole process, they are now locking the whole database. So if you have two different services which are talking to two different databases, you can actually have two concurrent writes, which essentially meant that if we wanted to scale out, basically wanted more throughput from databases, we had to add in another shard, add in a new machine, which can actually handle more requests parallely. One other thing which we noticed was the application was actually running under replica safe mode. So you have a lot of configuration for drivers when you actually connect your application to your database. So since it's a no SQL database, it's an eventually, it gets consistent eventually. But you can actually force your driver to have a replica safe write, which means that whenever you add in a write, it waits first to write on to the primary, and then it waits that the secondaries replicate, and then it gives you the response back. So we were being a bit traditional, and we were actually using replica safe mode, but it was not helping us out because it was taking a lot of time for us. So we thought probably we'll deal with eventual consistency and in our code. Yeah, so then we moved on. We fixed on to all these things. We corrected our image processing. We added a new shard. We added more power onto the front ends, and we got around 500, 600 requests per second. But still, we were aiming for 2,000. In the meantime, it was a small team, three dev pairs. We were working on new functionality. And there was a story which came in that, obviously, you want to authenticate. You want to add security to your database servers. And we thought straightforward, we added authentication. And we did a second performance run after adding some new functionality. And we figured out that the whole Mongo process was actually bombing out. It was taking too much CPU just by adding a username password, which was very surprising for us. We raised, since we were working very closely with Tengen, so this is around a year and a half back. We were working on 2.1. So we raised this issue with them, and we figured out that it was an issue in Mongo. We were fortunate enough to get a build back in a couple of days. They did fix this issue. So basically what was happening is Mongo was not able to manage its connection pool with authentication turned on. So there was some problem going on internally. So we got that, and we were again back on track. Third performance run. So we continued on our feature development. And we had a new feature wherein we wanted to. So basically people went on to the site. They added a bookmark onto some of the recipes. And they could also tag certain recipes with some categories. And the requirement was straightforward. You need to actually show with every category the count of the number of bookmarks which are there for a particular category. So we thought probably it's simple enough. We added that, and we did the third performance run. And we figured out that all of the other scenarios were working fine, but there was one use case, the count query in the API in the Mongo driver API, which was taking too much time, 55 seconds under load to return the response. So we started looking onto the code at what exactly is happening. Started looking onto the forums. We asked the tangent guys, and we figured out that Mongo basically uses non-counting B trees as their index data structures, which means that if you have to run any aggregate queries, it actually has to go, traverse the tree and calculate manually the count and then return you the response. And if it's a huge response back, it actually takes a lot of time. So we again reported this to a tangent guys, but since it involved a lot of changing their indexing strategy. And so they fixed this, but it was fixed in version 2.3.2. So after 2.4, the stable release, you will get it fixed. So as a result, we started maintaining the count with every category. So the solution left to us was whenever someone added a new bookmark, we actually used to maintain that count locally and we used to display when we fetch it. So this is what our final production architecture looked like to scale up basically for 2,000 writes. We started off with three front end servers. We ended up with 12 front end machines, six back end machines, and three Mongo Shards. Final throughput, we were basically able to handle 10,000 concurrent users doing 2,000 writes per second. But we thought, okay, we've achieved our goal for 2,000 writes per second, but let us see what is happening when you go beyond 2,000 writes. So let us see if we wanted to add or we wanted more throughput from our DB servers or we wanted more write throughput. What should we do whenever, if it's some anticipated load comes in? So if we, what happens when we add a new shard under a very, very high load? So we replicated the same performance runs again and we added, we tried to add in a new shard which was under, to the cluster which was under, which was getting bombarded with the request. And so what our learnings was that it basically killed. So it went into a deadlock situation and it eventually killed out the cluster because when you add a new shard as the new thing that you get out of Mongo is the auto sharding feature. It tries to balance your shard which means that when you add in a new machine it tries to balance data between your shards and it starts your data transfer between machines which means that it will add in more load onto the existing machines because they'll have to now move the data onto the newer shard. And also it clogs the network because there's so much of data transfer happening between the machines. So probably if you want to add a new shard you should probably have decide on a time of for doing that. So that was it. Any questions? You said that the MongoDB's previous indexing was not good enough to do aggregate functions. Yeah. So aggregate means like we were having so in terms of even so we thought that probably we are indexing a particular field and if you're doing a count on an indexing indexed field we should actually get the response really fast. But unfortunately because the way it uses the indexing structure it was not really aggregate friendly. So simply doing a count query was taking a lot of time. So you're saying when you're executing the count query the execution time is like order of n. Yeah. Yeah. So it's basically so in terms of indexing it has to actually traverse down all the index or whatever your results at it in your index and then do a count on every one of them. So how did like what is the improvement in the new thing and what? So they've started. So now they've started using in the latest version they've started using a counted Btree which means that for every level of the Btree they are also maintaining the count of the child elements which means that if you want to run any aggregate queries it will be much, much faster because it's already maintaining that internally. And what was the last data structure they were using? So it was a non-counting Btree. You mentioned about the database locks that happens during the writes, right? And that's still the case with their current version. Yeah, exactly. Does Mongo have an internal resource management strategy or anything that will take care of how long a database gets locked, right? I can write a quite a bad query and it's just continuous. Does it have anything like that? So you have explain on the queries, you can run. Yeah, that's on the application side but does the Mongo internally do anything? So Mongo gives you an Mongo monitoring console. So it gives you which database is basically you can monitor that. But... Internally... So what Tengyan guy says like whatever it preaches is that if you want to scale out you add in a new machine. So that is what, so that it doesn't lock into or doesn't achieve... Okay, that particular instance. Yeah. Cool, thanks. The last question maybe. Did you evaluate any other solution apart from Mongo because numbers like... So we did evaluate Cassandra but for our clients one of the biggest reasons were they wanted long-term support for their application. And Tengyan guys were themselves actually supporting it. And so they also do consulting for different companies in terms of teaching them or doing your API design or database design. And then also for long-term support they help you. So one of the biggest reasons for our client choosing Mongo over any other was basically they wanted a long-term support for the database. But DataStacks tells the Cassandra support. There's a company called DataStacks who support Cassandra. Yeah. In sort of similar manner how Tengyan does for Mongo. Yeah, I didn't get that. There's a company called DataStacks. Yeah, there might be a couple of companies which are doing that but for... So they evaluated that and they thought that probably the company's built it and is supporting it is better than some other company who's actually supporting a third-party database. So we... So yeah, so we started on... So for them so they wanted to... One of the biggest reasons were they wanted to scale out horizontally. And they wanted automatic fail-overs, which is... And they never wanted to add in any application layer which handled failures onto database. And Mongo was actually giving that out of the box. So if there is any failure of machine, even the primary as I was telling it, it goes down. It automatically does a reelection. And one of the secondaries is promoted to primary. And all of this is transparent to your application layer. So... Okay, wonderful. So he's around. So let's talk to him and discuss more. Round of applause for Sushit. Thank you.