 So how many of you have used N-Linux before? A third of N-Linux? Okay, actually how many of you have not used N-Linux? How many of you have not used N-Linux? So my guide is easy, I mean. So, this is about me. So I work for Adequity. Adequity is a mobilized platform. It's actually a product of Prodigy.com, which was a search engine. Not anymore. So what we do is, we are a mobile ad aggregator. So we aggregate ads from a lot of ad networks and serve it to publishers. So in essence, we serve ads. So typically, the job is coming in, we have a lot of requests coming in. In terms of unlike, actually the request that we use, it might be that on the page. So it's a job to handle a lot of requests. So that typically might make sure that there is no... This is the basis. So it was started sometime around in 2002. So in 2001 we started, which was a request for Modproxy in Apache. And it wasn't serving his needs, so he started writing Modproxy. Then he found that it's, I mean, by nature built first aid. So he kind of... So that is when he started. Sometime around, I think somewhere late, somewhere 2002, he started. And around 2004, he had the initial drafts for the project. And by that time, he had started running it on Ramka.org. And he was a Russian search engine, pretty famous for Russian Russian. So he was right around it. So that's why he ran it close to running. So the main branch of NNX is to address the C-10K problem. So C-10K problem is basically to handle 10,000 concurrent requests. So it's a... The job of handling 10,000... So when I say 10,000 concurrent requests, it's being handled by a single box, not having a array of boxes and you handling 10,000 concurrent requests. So it was actually built to handle it. So it addresses C-10K problem. And that, it does very efficiently. There are quite a few... So there are quite a few open source, like tornadoes, people have heard of them. And then there is Cheroling, which is also getting a lot of momentum now. People have adopted it after Apaches actually said that I think 25% of the top 10,000... I mean, high-traffic websites use engineers. I'm not sure when that was collected. It should be more... Now, after 10 years, now our engineers has pointed to a company now. So the C-10K problem is, as I said, concurrent... So this, if you've used Apaches, how tough it is. It hardly scales, I don't know, a few hundreds. If you come in, if you could scale for a few hundreds, then you've done a really good job trying to make it work. So the reason being, it's IO. So IO is always a bottleneck. It might be a fire IO or it could be a network IO. So Apaches' model of using a threat for connection never scales for, I mean, let's say, very high-traffic websites. So sites like... A lot of other sites, I mean, started using it. The main reason is Apaches can't scale. So if you want to have it, you'll have to have an array of Apaches servers, which again is a lot. You'll have to use a lot of machines. How this engineering handles this is using the E-Pole mechanism. So selected E-Pole are ways to actually find which file descriptors are active. So select mechanism is slightly... It takes a lot of time to use it. Apaches uses it, a lot of other stuff uses it. E-Pole was started, I think, 2.4 kernel, I'm not sure. I think 2.6 does support E-Pole. So what essentially it has is select takes a match file descriptor as one of the arguments. And it has to run through all the file descriptors and see what is active at any point of time. And this is a lot of gas. So every time an event is active, it has to run through all of it to find which is active. What event and what is active. Whereas E-Pole takes in just which are the file descriptors you want to consider and takes in that. Now the advantages in select is you don't have to pass the file descriptors. In which case the memory requirement is slightly less. But it kind of outweighs the performance gain E-Pole gains. So that is how engineering actually scales better in terms of being fast and able to handle concurrent 10,000 connections. So small trivia, we at the moment handle roughly around 5k plus around 6k requests per second in a single box. And recently we moved to two boxes just to have a prevent single point of failure. So the URL which I mentioned is a very good resource for people who are interested in systems. I mean it's more like you need to read it if you are interested in finding out what are all things you need to know to scale in a system level. So these things are at a high level why engineering is preferred among people. So the main reason is the layer 7 load balancer. So actually when we started around 2008 we were looking for some load balancer then. We had options of LBS, it's called the next one. It did work out but the main problem was it was layer 4. So we couldn't do it, so you cannot load balancer the URL level. Let's say you want staff users to go to a different box and all the staff couldn't do it with LBS. Otherwise LBS is also a very good solution for load balancing at a layer 4 level which is at the TCC level. And it has the reverse proxy. So basically you can come in, so mod and scope proxy of Apache is kind of inbuilt into engineering. So since it was kind of, I mean it started with how they wanted engineering, it hasn't been built. Other one is embedded Perl interpreter. This is a very nifty feature if you want to do something, I mean something which engineering doesn't support with this. Usually it will support most of the things you want. But if you want to do something else, let's say pass a piece of text or something else and you are familiar and you are comfortable with Perl, you can write a piece of Perl code into it directly into the engineering response and have it pass it through your back ends and however you want. So in the use case, as I said, let's say you want to do something that you don't want to pass it to the application server to do it. Let's say it's a very trivial job but it's being done every time. So you want to, and it is very necessary before being sent to the load column server. Let's say for example there is a piece of text but you want to pass the text and see based on that and you want to do it. So yeah, so that kind of things you can write, I mean if you are familiar with Perl, you write a piece of Perl code, put it there, it does a just in time compilation and pass it there. Or you can have regixes, I mean let's say complex regixes, all the stuff you could have. So you are going to write, I mean is, that is just, I mean you are going to write is supported natively in engineering. So it's supported VCI. So Perl is like you can have proper Perl code. Let's say you want to have a particular module. Some, let's say you want to have the application itself. You have a backend of CGA, Perl CGA application. But you want to have a piece of it inside your engineering. So one example is let's say smart detection of user agents. So in R, I mean this is, it has actually a reason why I'm telling this is we have this case where we wanted to find out the device of work. I mean from the user agent we need to find the device. So and for some cases they say you want to find it whether it's an Opera Mini or not. These kind of stuff. So these kind of stuff you can do directly with Perl. I mean it's pretty simple. You can read the header, do it and based on that we can set a variable. And if the variable is set you can send it to a particular post or do a particular thing. So this is a very simple feature. And the other one is on the fly binary upgrade. This we often use when we have, when we do change, when we push to the next version, we do this. So the good thing is you don't have downtime. So what typically happens is you have an engineering's binary running and you send a USR2 signal. So when you send a USR2 signal to the master process, what it does is it creates a copy of the binary with the PID. It says PID dot the binary. And then you start executing the new binary which you have given to it. Now at this point of time both binaries will start accepting connections. Even the old binary and the new binary will start accepting the connections. Then you send a win signal. When you send a win signal to the old binary or an old master process, it will gracefully shut down all the worker process. So when I say gracefully shut down, it's basically whatever request which it's got until then are serviced. It stops accepting new connections and it shut down all worker process. And since it's shut down all the way, your new process is live and it goes with that. You almost have zero downtime. In other case, I'm not sure if there is anything like that. I think you can do it with some kind of shell scripting itself. But this is something which is built into NGX. The other one is concept changes. So let's say you want to change concept. In Apache, I think you have to restart it. If you do change in HTTP.con, you have to reset it. Whereas I think the ones which are in the document, I think it's dot rewrite or dot HTTP or something like that. It's got HTTP access. That, if you change it, I think you don't have to do anything. But the main HTTP.con, which is in the HTTP or other one, if you change it, you'll have to restart it. In which case, it has downtime. Even if you do a start and talk, it still has a minimum downtime. That is avoided here a lot. Especially when you're handing a lot of users and you want to see a very good experience for users. This one which we were telling is the PCRE support for rewriting URLs. So this PCRE module is built into the NGX. So any kind of regates match, you want to do it on the URL and rewrite it. You want to do internal rewrites. You can do it with this. And of course, to sum it up, it's super fast, lightweight. And the low memory footprint comes because of the ePoll mechanism. It doesn't follow the single connection per thread for every third connection kind of module. And it's like feature rich. Now, there is a saying which goes, which says, I think Apache does six things, but everything bad. But NGX does all these things. I mean, five things out of it, but does it way faster than that? Yeah, but it actually does it. So this is on the overview of the config. So the config is actually, it's a hierarchy model. So it has a HTTP scope, then it has a server scope, then it has a location scope and a upstream scope. So these are all the contexts actually. So HTTP scope is bad directives where you will put to have, I mean, things like number of worker processes you want to have or a number of connections you want to have. So this is for the whole engineering server which you're running, the installation you're running. The server context is used for virtual servers. Let's say you want to have to run a server on port 80, and you want to have a HTTPS and supported port, another one. Or you want to have an 8081 also for testing purpose. So you can have everything inside the same con and have, according to it, whatever conflicts you have. So everything which goes into the server block. So since it's a hierarchy module, so HTTP will have multiple servers, server will have multiple locations and location will have multiple upstreams. So it will all, I mean, the hierarchy followed by any directives which are applied at the HTTP will be inherited by the blocks inside. So yeah, so server block also can be used for handling things like, let's say, it's a web host. You want to have, you want to give a single, you want to have let's say one server, you share it with multiple people and with subdomains. So then what you do is you just have multiple server blocks and based on which domain is accessed, the server, there's a direct with one server at its own name. So based on which domain is accessed, it will go into that server block and things inside that is configured or used, executed accordingly. So location block is like, which defines what the URI should do. I mean, let's say it's a slash users. When it has slash users, what should it do? To which backend? You want to set some features and all the stuff, you could do that. So this is like the route in Node.js people have used. So it's typically like that. Just that application code doesn't come here, it just says what each one of them has to do. So if you want to do a REST API ping at all, you can set it up here and accordingly have different backends. It's very easy to do. The upstream block is the block which says which are the proxy servers. So it says A, B, C, D are the proxy servers and you put it in this one. So when you get a request, so in the location block, you'd say if it's a slash content, go to this upstream block. The upstream block is the name of the upstream block and inside that you'll have four servers. So it'll go to one of these four servers and you can define how to comment what mode we use to go to the servers. That I'll cover up in the next slide. So these are a few directives which are very important when you consider NGINX. So worker processes, so NGINX model is you have multiple processes. One is a master process which kind of receives all user signals if you want to pass it and then you have worker processes. So anything which is passed to the master is passed to the worker automatically. The worker process has to run as a root if you're running it on a standard port less than 1024. If it's a non-standard port, you can still run it as a non-standard port. So worker processes, you set the number of worker processes here. So you say worker underscore processes, let's say 96. So 96 processes will be spawned when you start NGINX. Worker connections on the other hand tells you how many connections you want to handle per process. Now these two settings actually are very important for you to know how many concurrent connections you can handle. So the number of concurrent connections which you can handle is it's a product of number of worker processes into the number of connections. So if you know if you can expect this is the kind of traffic I'm gonna have this time so you can set it accordingly. On the other hand you can also set it to a very large thing but since it's a number of processes it's going to consume a lot more memory. So every single process is going to consume some more memory. So if it went on a requirement basis you can change the config accordingly. Worker R limit no file is to set the number of file descriptors for the open sockets. If in Apache and all for that you'll have to do it at the OS level. I mean you'll have to write it in I think slash crop or somewhere to set the number of open descriptors. Just you can do it in the engineering transfer kits and do it. So worker R limit will be roughly be the same as product of worker process and worker connections. This is the number of open file descriptors at any point of time. So keep alive is a HTTP direct command. It's a header which is supported since I think HTTP 1.1. So it basically says don't close the connection in the client side. So because of this the client when I say client it's a browser. Typically it's a browser. So the browser actually holds the connection so that it can use it for another request within the same page or other. I think there is a browser constraint as to a domain you can have only two active requests, two active connections. So keep alive request on the server side. It says number of active connections it can have and the timeout it can set. So when you say 30 seconds. So these 100 connections will be held only for 30 seconds. The moment it passes 30 seconds it will start closing the connections. This is very important because let's say you want to in the case of ads which we do you want to actually show the ads as fast as possible. You don't want the user to wait. So in this case since the connections are already kept open it doesn't have to do a connect tagging. So this kind of saves a lot of time for the user to do. So it's only the read and write which will happen. It doesn't have to do so in Firebug if you do it will show you the connect time and then the read and then the write I mean response time how much time it took to make. So connect time will not be there if you do if you set the keep alive. Auto scale worker process I think there is no direct way to do it. There are other ways to do it actually. So it might not be that. So it depends. So you need to find out where your bottleneck is. So typically it will become that engine might not be the bottleneck at all. It might be that your back ends are bottleneck. So you have an array of application servers that might be blocking a lot of request. Engine might be pretty cool. I mean they'll be handing it very fast. So auto scale as such it's not there. But what which I'll cover in the next couple of slides is there are ways to find out how much time it takes to respond to a request. So you could of course write a kind of shell script or something like that to find if there is anything like that. It takes the latency is high. You can do automatic conflict change and do a rainbow. So it's all these can be scripted scripted. So that way. Yeah, it's the same. So since the conflicts can be... It doesn't require a restart at all. Go change the conflict and do a kill hop. Hop is a hang up signal. We pass it to the master process. Read the new conflict. If the conflict, let's say you have a syntax error or the conflict is wrong. It will not load the conflict at all. It will use the old conflict. So it's like, I mean you can do whatever kind of auto scaling you want. In fact, I think probably think this same kind of mechanisms which Amazon's auto scaling is. I mean, it's typical kind of things where it automatically changes one and have more units. Well, it depends on the problem which you are addressing. I mean, the kind of request if you are on the cloud and if you have, let's say, ability to add servers automatically. Or if you have, let's say, a lot of resources in the system which could be used. You could definitely use it. This usually kind of changes when you have a lot of static content to be changed. So only then this will be used. Otherwise, I haven't seen, I mean, auto scaling, they haven't done. We kind of do it. We check it once in a few days. See if there is a problem. Typically it doesn't happen that way. Kind of scales. At one point you'll see that there are a lot of things. Then you go and check where the problem is. And you find it and you kind of change it. If it is 96, you make it 120 years. So it kind of scales for the next few billion requests. That kind of thing. Personally, I haven't done auto scaling with this thing. But though we do something else where we kind of change the weightages to each of our backends automatically. So based on, it's a very dumb way to do it. But that was the thing which we could do it with a short amount of time. What we did was we kind of go check the resources on each of these backends, collect data based on what is the load on these kind of servers. We kind of change the weightages here so that the other one gets it. It's a poor man's fair balance. So this is the upstream block which is the upstream block. So this is how you configure upstream block. You say upstream. The backend there. It's the name of the upstream. And then you say server, and you give the ITs or the domains of whichever host you want to have. So here in the example, I've given four hosts. So if you see, so you can have hosts running on whatever port you want. So when you say main port three, so let's say you have six requests. The first request will be sent to the first server. Second, the next three requests will be sent to the third server. So it's a weighted RR, weighted roundup. And the last one says max fails three and fail time mode ten seconds. So this config, it supports this kind of config also where you can say the maximum number of failures a particular backend can take. So when I say failure, it is actually defined by a property called proxy next upstream, which says what a failure is. A failure could be an error or a failure could be a time mode or it could be a particular error code, HTTP error code from your backend. It could be something like a find out tool, get me timed out. So generally let's say I find out, let's say they have changed something and you're getting an internal server error. Any of these could be configured there and that will be taken as a failure. And here what it means is it says it can take maximum three failures within ten seconds. So within ten seconds, you have three failures for this server. Then what will happen is this server will be made in operator. So it will not send any more requests to it. So automatically it starts using only the top thing where there are no, you can set the same all of the servers to. So here there is also this one thing called keep alive and a number. So here what it says is the number of live connections you want to maintain with your backend. This also helps, wherein you don't have to make a connection to your backend. So this was supported only recently. I think 1.1.4, which is like a year ago. Now it's like 1.2 or something to work with. This was supported very easily and this we've been using it and we found that this has a lot of benefit in our kind of setup. Yeah. It will actually re-initialize the counter. So it re-initializes the counter. So ten seconds the time won't. So you get three, let's say you have a burst of three requests and all three fails. It will make it inoperative. So it will wait for the ten seconds to elapse. After that it will actually re-initialize the counter. So then it makes it. So within ten seconds you shouldn't have three failures at once. So keep alive is just like the keep alive which I mentioned sometime back. This keep alive is for maintaining the keep alive request with the clients, which is typically the browsers, the users. This keep alive is to maintain it with the backend. So Nginx keeps a keep alive with your backend server. So your backend could be something like an Apache server or it could be a Rails or a Python or anything. It could be. So it keeps a live connection. So which means that it doesn't, you get a request and you want to send it to a backend server. It doesn't have to do a connect again. So there is already a connection alive. It will send it through that. So it saves a little bit of time. So here it says, maintain 2048 live connections with the backend servers. Somebody make a connection to do that. Yeah, so this is more connection. Do the pooling of the connections. Reuse the same connections for the next request. So and yeah, so you set up the upstream block here like that and then you say in the server block you say location when you have a slash let's say wwwabc.com. So the slash, so I'll, so this location block actually tells which, this is like the which URI, URI which you have got. So it will match this and it will say it says, pass this request to the backend. Backend is the name of the upstream which you have set. And it goes into here and then it uses one of the models whichever you set. So which is here it's a weighted file where this one will get slightly three times more than the other ones. Yeah, so here in this case since I haven't put any match fails, even if it fails it will keep still set the secret request. So that's why you have to put match underscore fails. If you put that it knows that there is a failure there so it will stop, it will make it inoperative. Yeah, it will, it will make it inoperative. Basically it's like commenting out this. So it's like commenting out. So bring it back when the time is elapsed. So the 10 seconds is elapsed, it will make it actually initialize the variables again and it starts again. Again if you get it three to come in when you see it again make it. So there is also another directory which I will put it in the slide where you can hard port a particular server as down. So when you say server a.d.c something like that say it down which means that that server is down. So it will never send it across. Like commenting out but you write it down. So this is used when you want to do some kind of automatic configuration changes. So let's say you know that I mean you have some kind of heartbeat mechanism separate and you kind of ping each of the upstream servers and you get responses and based on that you know that something is down. Go ahead, chain the conflict automatically saying this is down. There's also another one which says two servers and you can give it and say backup. So the backup servers are used only when these are not active. All of these active. Otherwise it is always there. So in the other case what happens if you put all the servers here all of them will be used no matter what. So it is kind of bad and you want to do easy to run out. So it will be built for the iOS by sending to it even though the other three servers can just handle it I mean easily. So you kind of say the servers here and then you say it as backup it will never be used until all three are famous. So this is the name of the backup which is where to pass your connection to which is this. You can only put a name. Responses. There are four servers. Say there are 50 servers. Yeah. Yeah. And say it will work automatically. Yeah so that it can be so you have this include mechanisms where you don't have to have the whole config in a single file. So inside this on this you can put a include a.con inside that you can have only the single file. So inside this on this you can put a include a.con inside that you can have only the server for the name I mean the items. So there is no direct way of doing it. But it's the config supports variables that are used to do which one of them is like down. Then other one less let's say you do max fails itself of one and a phase time out of let's say one request per second. What happens let's say you start a server or you don't have the server. You've shut down the server. So it will since it's max fails one it stops sending any request to you don't have a problem at all. So it only the other ones will be used. The moment you switch it on it will send it on one more and it will know that it's active and it starts it. So wait is the so the difficult way to do is the round robin which is try this one the next request comes and it goes in here if you don't have the made it goes in like this. So the first request will again go to this one. The weighted round robin is you're giving weightage to it. So let's say a particular you have heterogeneous set of machines one server let's say has multiple and a lot of requests. So then what you do is you set the weightage 3 which means that for every let's say first request comes second request comes then the next 3 requests will be sent to the server because it can handle more. Yeah. So earlier I said you can change anything you want at runtime in the conflict and send a hub signal hub hub is a hang up signal to the master process and reload the con. So there is no it's a master it will be running or it will also say master. There is a PID so PID file will be created whenever you start an engine X that also you can it's actually set in the conflict what is the PID filename you want so you can just say kill hub cat that PID file in tic-pactics that will also put a new conflict here. So you have log levels you can set it in 4 debug and everything if you go to the log you can say that either if the conflict is wrong it will say the conflict phase to reload and it's using the conflict to say that I've received a hub signal and I've reloaded the new one. So this round-robin versus a farm balancer so not always does round-robin actually work so the problem being it kind of sends a request to every other backend always it keeps round-robin I mean doing a round-robin but sometimes it might be that one of your backend servers you do some request that's a DB bottleneck it's doing a lot of connections it keeps queuing up the connections to it it will not wait I mean it will take the connection and it's going to do it queuing it up now this is bad because a set of users will see that it's always I mean it's taking a long time and Nginx is not helping much so there is a third-party module available Nginx I mean there is not yet available natively if you compile the third-party module with Nginx you have this thing called fair where fair actually says now when you put fair the server knows the number of requests each of these is taken so the next request is given to the least busy bucket so if 1.2 is got let's say 5 requests and it's still processing them and 1.3 let's say it's got only 2 requests so it gives it to 1.3 instead of giving it to 1.2 again in the round-robin method it will be like every single time it will be like doing a round-robin of these it kind of maintains a each one is got and how many I mean whether the connections are closed number of requests are the responses are got it doesn't show a file there is a way to find it out I'll tell you there is also another thing here which says no round-robin the no round-robin is like so in the normal way to do it is when you say fair it kind of goes to the next least busy server when you say no round-robin let's say the same I mean has finished processing the request it will take it out again so we can kind of send it to the same server and so you kind of know what level your backend can scale so you can know what is the total number of concurrent connection of particular backend with scale so this is proxy next upstream which which is talking in the failure failure underscore time zone max fails so when you say max fails here the failure is defined by this the failure basically says your error so error here means it could be anything error it could be something like a final error or final 2 final 4 whatever you have it could also be just a time-out particular so it can take things like error or just say time-out or invalid header let's say the backend responds with an invalid HTTP header then also you can say that's a failure error code HTTP underscore final 4 so only for final 4 it is considered failure no it doesn't include I just so just to emulate this you put error so it's like it supports all of this have 10 more minutes okay I have lot more slides I'll just run it through so add header is to add header to the response so when you want to send a response to the client and this so what you could do is you just say X some X who is the who which backend it is and send that IP and send so based on that you kind of get to know which backend actually serve this response and cache control can be done directly from NGX the proxy set header we really don't know so this cache control so you can have this you have this CPV or something like that I never use cache control as is like always dynamic what we do use add header is for debugging purpose to know which backend it is so basically it's nothing it can take any header and any value so this is the header name and this is the value it is added to any proxy set header the proxy set header actually is used to send a request header to the proxy this feature is I mean we use it a lot because let's say you want to enable a particular feature in one of the backend this can be used for let's say you are developing on an experimental feature in your backend and you want to do it only for the you want some live traffic to see how it kind of behaves so what instead of going and changing the config there and enabling in the backend in your application server you write a kind of code where if you read headers headers are cheap to read because you're not reading the body of the request and you say if this header is set or if this header has this value then I could execute this feature for so you can set the headers in the engine itself so you have multiple backends you want to do this at one point in your engine you said you said this proxy set header so in this example I have this proxy set header X I mean you have a feature full and you set it to one this header can be read by your application servers and if it is set to one you execute a piece of code if not you don't execute it so you can change it you can toggle the flag in engine X and based on that you can kind of see the behavior of your experimental line graph this is another good thing general case I don't know a lot of people might not use this so the thing is if you want to draw beacons the beacons are to know in the HTTP world to know if there's an event happening to another third party server the easiest way is to send an image tag so in the image tag you say image src equal to something now if that is being called it means that a particular event is under now this src thing that you use is a big resource and it should not be a 404 with a 404 it's a bad kind of design so for that what we do is we draw a one by one pixel so this is supported by natively by engineers so it has a from the memory it kind of serves a transparent gif when you say you have a request saying abc.com slash beacon and you put empty gif this pixel will be kind of served it doesn't get terminated it serves a it's a response ascent the only thing is and the other way you would have done is you would have had a file in the file system when you would have read it you should have configured to a small size should read it this is in the logs you have it in the logs so you have in the logs saying slash beacon from so and so so when you do crunching of your logs you get to know what events and what actions can be called by which this is one more feature which is used to set to trot to the number of request set per zone so this is when we use it a lot during our search times we had a lot of bots crawling in our search engine that time to mitigate the DOS attacks what we do is we do this we said limit request zone and you set binary mode address and then you say zone name this is 10 m s s so much memory and you say the rate it says one request per second so it can take a maximum zone name so you can configure multiple zones according to whichever blocks you want and say what is the zone name and accordingly trot to the request so anything above let's say you get two requests per second it will do a delay it will not do it the burst is actually let's say you get four requests at any point you set up burst level then let's say you get four requests per second for all these four requests it will delay anything above some unavailable error code this is another config which actually tells you what is the concurrency limit the other one tells just the request but here it says concurrency at any point of time let's say a particular domain can handle let's say it can have one connection at a time so some kind of watch what it will do is it will just pawn threads from somewhere and it will try to print down a server getting videos also you say the binary remote address whatever it is and then you say limit connection one which means that it will allow per host only one connection at any point of time only after the connection is closed it will allow another connection yeah so the IP block that is a separate thing I mean it's nothing to it you are talking about IP block so there you can set there is something called deny and allow so in deny you can set up let's say you know that these IPs are some kind of bots you can put it on a list and you can keep up waiting there is a deny and d6 set up you could do that you can put this in a particular block and say a deny block then there is this this is one sometimes back someone is asking this sub status so sub status actually tells at any point of time what is the status of when it works so say you've done location ng-stat take sub status on and if you access ng-stat on your so here you can actually put allow a particular IP let's say you are local in your office IP or something like that in your production IP so that only from that IP you can access this location so when you access this location it will tell you number of active connections number of total connections handled after it started in this case it's around 1.6 16 million actually and the total number of business so at any point of time it will also say what is right now number of in the right state in the right state and in the waiting state so you keep refreshing it you get to know what it is that's so you can actually kind of have a script which can take data in peak times every 5 minutes and you can have a graph plotted to find out what is another one is the map directive the map directive is just like I mean it's like the hash map concept where you say map these are inbuilt variables and you say new is another variable so here you say basically you are right if it is slash aa then I want new to be set to aa.domain.to and bb if it is slash bb and so and so but in the default it's like a switch case you do all of this and you say default if none of this is there default is this now in the server block you can say rewrite dollar new I mean this is it's a so when it it will get redirected to the actually there should be a space here so this is very helpful if you want to have let's say domain parking and all that stuff this is another very recently used I mean release feature called split clients this is used for A B designs let's say you want to test different colors or designs to a particular set of users you say a split guy and you can use that can be hashed they're cooking or remote so these are all inbuilt variables so remote and inbuilt will be replaced by the IP here and you can say 0.5% of the request will go to dot menu it will be set as dot 1 and 2% is dot 2 here you say index index dollar variant which means that 0.5% it will be replaced with index.1.html so that 1.html might have different CSS or anything and you want to know what kind of perception people have then sub filter is another filter it's a directive to let's say you want to change something in the response and you've got a response and you want let's say copyright is there and you know that this is a useful thing where people don't update so you want to change it before going here and changing every different file what you can do is you can write a request in sub filter make it always whatever all responses anything that you go will be scanned through this and it will be changed and sent across error pages just like Apache custom error pages can be handled you can set whatever you want to do let's say I would make it whatever then there are this I'm not covering these these are very important characters if set and rewrite if it supports if condition so any variable you can check if it supports one year and you can do a set set is like assigning a key value value to a variable and rewrite is for internal rewrite and you want to do a redirect based on electrics and there are built-in variables dollar r underscore parameter let's say you have slash uses and id control 243 and now if you want to read this parameter and according to something what you do is the engine is then according to any kind of same way HTTP any HTTP headers it can be all underscores will be replaced by hyphens all hyphens will be replaced by underscores and these are the variables which I was talking about so this and this are very important to know what is happening so you can use dollar request time which actually tells you end to end time of how much the amount of time is taken then upstream response time is to find out what stream you are taking so with your upstream server address also long you can kind of get statistics which upstream server is taking there are other lot of modules XSLDE and these are all inbuilt modules so some problems are this you don't have like Apache where you can load a module then the documentation is bad now it's good since it started with the question thing it is always a bit of bad documentation there is a very good community IRC there are people who have answered all your questions sometimes we don't even do a Google we go ask IRC they kind of answer so to the point it's like pretty