 So, I am going to be talking about how to do scale without doing scale from something that I got involved in a month ago. And if you haven't guessed yet, this is what I am talking about. How many of you guys have seen this website? So you use this to send an email. At least a million people did that and that was not what we were expecting. We built a site thinking we will get 15,000 people. So building for 15K people and getting a million users gives you scale problems that we had two days to fix and we obviously tried our best to not fix it. So I am going to tell you how we actually managed to run a campaign that large. I can't tell you how we got so many users because we don't know that. What I can tell you is how we survived with so many users. So to start with, let me give you some stats so you can see what I am talking about. If you go into the Save the Internet campaign, obviously it's completely dead now, nobody is on the site anymore. So I am going to go back in time a little bit and say, let's pick from the 1st of April to the 30th of April, roughly the period on which the campaign ran. So what happened here is on April 1st, roughly, we had a sense that we should be doing something about this. I am going to drop the resolution a bit so you can see this a little more clearly. So on April 1st, we started doing something about the site. The site went live on the 11th of April. So on the 10th, it was 0. On the 11th, we had this many users. You can't see the number clearly, but it says essentially 160,000 sessions, which is very, very normal for most websites. And then on the next day, on Sunday, it went up to almost 500,000 sessions and stayed there. And almost everybody who went to this site also sent an email. If you look at the unique count, it's 2.5 million users went to the website, mostly over the period of one week, or actually two weeks, but most of the traffic ended at the end of one week. So now 2.5 million users to this website sent about 1 million emails. There is a fairly high conversion rate. And we had two problems. One, how do you make a website that doesn't crash when it gets hit so suddenly, especially when it was built by people who were not building with a large team to manage things. We had no team. It was basically one intern working on the site who's not even in this room today. It was run with no expectation of what would happen when we had so many users. The only thing we knew when we started working on the site was that we didn't have the resources to do this seriously. So when Nikhil Pawar, who initiated the campaign, called me and said, we need to build a website to help people send email, my first reaction, honestly, was, man, I already have enough work. Don't give me more work. So at that point, it was about saying, how can I do this without doing too much work? And so we made several design choices in how we built this site. One we decided we are not going to write any custom code. We're going to use everything off the shelf. We are going to use free software, free as in free beer, not free speech. We don't have to pay for stuff. If it's free, we'll use it, which is how we end up using Google Analytics, which is how we end up using Heroku, and so forth. And it had to be as simple as possible. So one of the things that we had to decide fairly early on was how we were going to send email. So while there was this concern that if you say, tell people, here's a template. Please send the email by yourself. They'll most likely not do it. If you say, please put your name and email address here and hit send. And we will send an email for you, which is essentially what most petitioned websites do. They say, give us your details, and we'll take care of the rest. Of here was it would not be taken seriously. And there was also the concern that that means we have to handle the email sending part. And that was the first thing I was doing there, thinking, if I had to send email, what would it cost me? It's one thing to say I'm going to sign up for a free SendGrid plan. SendGrid, I think, does 200 free emails per day. So that was the first thing I was thinking, man, I have to buy a mail package from somewhere, whether it's AWS or SendGrid or Mandrill or whoever else. So the first thing we figured is, look, we're not going to bother with that. We don't want this hassle of running a mail server. Let people send email by themselves from their own computers. And we'll figure out how to make that happen as smoothly as possible. Now that choice also had an unexpected benefit for us, which is that this website that we built is a static page. There is nothing behind it. There is no web server. It's essentially just a static page that's being served to everyone. And everything happens on the client side in the browser. So now this is a choice that most of us don't really consider when you're building sites. And we keep thinking of what kind of servers we need to manage scale. What do you do when so many people will hit you all of a sudden and so forth. So in our case, we did none of that. Lefty does static site. The problem was, how do we know someone has sent an email to the telecom regulator? Well, let's just ask people to bcc us. So we made an email ID called networkalityindia at gmail.com. Once again, free services. First, we consider whether we should have something at, say, the internet.in, you know, and say, look, this looks legit because it says, say, the internet.in and the domain. Look at each other and figure out, man, that means setting up a mail server. Not going to do that, get gmail. So made a gmail account, told people to bcc it, put the site up on a Saturday. That was on this day. So Saturday, at roughly about 1 PM, that site went live. And we tweeted out to a bunch of people and said, please go submit. The website is live. So within an hour or so, we had about 1,000 emails already sent. So that was the point. We were looking at it and saying, oh shit, people are actually using this stuff, you know. This seems to be enough public sentiment that people are sending email to each other. Now this was also the point that All India virtual released their video. I had not seen the video before they released it. Someone else had seen it. So obviously, there were people working on different parts of the campaign who knew what was about to hit us. But we were sitting there, that video went out and I don't know if you guys have seen the video. You guys want to see part of the video. Who's not seen the video? There are people who have not seen the All India virtual video. That's incredible. Okay? So this is an 8 minute video. It will take up half my talk if you play it. So I don't know if you should play the full thing, but let's see part of it at least. I should have done mute on the mixer. Let's give credit where it's due. Alright, telecom operators have done plenty for this country over the last decade. They've allowed your chargers to send in dirty SMS jokes all day long. They've even tried to convince you that every friend is important. You know what? Every friend is not important. Some friends are Matlabhi and are only your friends because they want to play FIFA and you know the only person they know who has an Xbox. Tell your pretend friends to anyway. The point is if you're one of those three people in the country that actually likes your telecom operator, don't worry. Because we finally have something that will push you over the edge. Because right at this very minute, India's telecom companies are lobbying the telecom regulatory authority of India to enact a regulation in a way that will change the way Indians are using the internet forever. These telecom operators are going after something that's called net neutrality. Okay, so I'm not going to play the full thing. But right at the end is a critical part that got us really, really worried when we saw this. What you need to do right now is log on to www.savetheinternet.in. All you have to do is log on to the site, type your name and your email address and with just two clicks you can send an email to try answering those 20 questions. If you're on mobile, click on the link in the description or on the i-button on the top right of the video and tweet using the hashtag savetheinternet. And tell your friends and your family and anyone you know who has ever used the internet to do the same. It's worth it, we promise. No, no, no, no, no, no, no, no, no, don't subscribe. Don't like, don't, guys just send the email. Do what we said, you know, if you want to not be ripped off by a telecom company or your ISP, just please go ahead and do this. Just go, you need the internet. If you don't click you will not, this will not happen very often. 24 day print, 24 day print, tick-tock, tick-tock, tick-tock. Stop waiting, look at my hair, this shoot is at 4 in the day. Like seriously, if I can make this video at 4 in the morning sweating, you can log on to the website and send that email to TRI. I'm just going to wait here till you click. Just, I don't know where to be, I'll just wait. So, if you notice, that was a very unusual footer for the videos. Almost every video, please, you know, says, please subscribe, please do this, please do that. This one said, just go to the damn website and send that email. And they had 2.8 million views. So if they had 2.8 million views and we got 2.5 million users, that's pretty damn good conversion rate, okay? So, now what's happened there is that Saturday afternoon, we put the website out, watched it for about an hour. Notice that we already had 1,000 emails in an hour. And then AIB decided to go put their video out by about 4 p.m. And by midnight, we had 19,000 emails that had landed on that mailbox. Now, if you know Gmail has rate limits, it only allows you to receive about 800 emails per minute or 3,600 per hour. From 1 p.m. doing 1,000 emails to midnight doing 19,000 emails, is clearly hitting the rate limit of fairly close. So we spend the evening looking at it and saying, man, this is getting some traction. What do we do now? We're going to start bouncing email any moment now. So someone had the idea and said, why don't you guys set up your own mail server? You can get a professional mail service that does not have such rate limits. And the rest of us are looking and saying, maybe we'll just make one more Gmail account and load balance it. So we did that. We did netneutralityindia02 at gmail.com and 03 at gmail.com. And put those on the site and just set it up to randomly select one of these three. And we watched the second day. By the end of second day, we had hit 100,000 emails sent. And again, we're looking and says, all three are bouncing now. What do we do? So we figured, okay, let's expand. Let's make more Gmail accounts. So we went up to 15 Gmail accounts. Third day, we were looking at this Monday morning traffic coming and 15 of them started bouncing. So what do we do now? Once again, someone said, why don't you guys set up your own mail server? You know, it's not that hard. You can just go to Mailgun or one of these guys and buy an account and write a mail there. And we're looking and saying, guys, we don't have any money. It's not like we're doing this as a commercial campaign. We just did this because one, we had no time. All of us are busy with doing something or other. And we're trying to make this as simple as possible. Second thing, the moment you start introducing technology and say, I will use my brain and build some fantastic piece of technology. Great, what happens tomorrow if your boss calls you and says, come back to work? Who's going to maintain the damn piece of technology? Because this is not a commercial project. This is not something that was being run as a coordinated campaign. It was a bunch of guys working in the spare time. So 15 Gmail accounts overloaded. What do we do? So let's make 15 more. So we made 30. By afternoon, we said 30 is probably not enough. So we made 40. And then said maybe this is still not enough. So we made 60. So at the end of the campaign, we had 62 email accounts. I happened to have a copy of their archives. So I was looking at the timestamps of all the mail that we sent. These are all the ones that we got BCC done. We got BCC on 1,050,000 emails. Total of 98,000 unique email addresses. So a lot of people just basically sent mail multiple times hoping it would convince, try a little more. And I've been trying to see if I can do some kind of analytics on this. But trying to even parse this many dates is insane. So there you go. So that's how many I've got. These are the mailders of each of the mailboxes. This is 60 GB of email. Let's see if I can. Sorry? Yes, it's just that you have to download it over IMAP. So it took us about four days to download all of this. And the guy who downloaded this happened to be sitting in Chennai. If he's on the live stream, hello. Hey, Shreyas. So Shreyas downloaded all of this on his computer at home. And then he said, OK, how do I give this to you? And he said, you can't just reupload it to us because that defeats the whole point. So ultimately he couriered me a thumb drive with all the emails and sent it. This will get to Bangalore faster than trying to do this over the internet. So as you can see, trying to even find out how much space I'm occupying is taking so long because there's so many images, a million files in there. But basically we did this. It cost us exactly 0 rupees. The only thing we spent on this entire damn campaign was on the website domain which somebody had already bought. This gave it to us. And it's clearly had enough impact that all kinds of crazy things have happened as a result of this. Now there you go, 61 GB. That's how much mail there is in here. So now one of the interesting things that happened is that after we did this, the regulator decided to do something that got a lot more people wondering what the heck was going on, which is they put all the email online, including the email address. And on the 25th, what was the date they put on? 26th, right? 24th was the Friday. 26th, 27th was the Saturday. On 28th, it all went online. So whoever was sitting at trial working on the weekend to put all of this online, I'd say kudos, man. Look, we didn't have that kind of patience. So they put everything online. So very on that Monday, everyone started tweeting about how this was an email market year's gold mine because 1 million active email accounts that know what the internet is and care for it in the last one week and all the email addresses are online now. It turns out that they didn't know how to put a website. The anonymous guys decided to take down the website as remains. All kinds of things happened. And the bunch of us sitting there and saying, what the heck did we just do? I mean, we thought we were saying 15,000 people sending email to the regulator. We'll get them to set up and take notice. It turned out to be a lot more intense than that. So now somebody built this website. Hopefully my connection will hold up and we'll see this. But basically this is the work of Anurag who's also not in the room today. Anurag lives in Pune. He works at Red Hat. He figured if there's going to be so much email online, then it'll be interesting to run through, scrape the entire database. So he downloaded 60 GB over from Try Again, not from us. And put up the site where you can type anybody's name or email address and you'll find the email that they sent to the regulator. So someone wants to try theirs. Somebody's email address. Come on. I mean, did he send it from his official address or his personal address? There you go. So if you notice, there's a little tag that says that it was sent from the campaign. So what happened was after Anurag built this website, we gave him a copy of a database and said, OK, you can mark off the ones that came from us. So we know who was not from this and sent mail to the regulator. If you look at this stuff, this is, I think, basically the standard template that we built. So it's part of our template. Obviously includes four numbers which are not supposed to be public. But that's a regulator. It's not us. We didn't realize anything. Now, some of the stuff that's interesting, not exactly scale-related, but interesting is, I think there's, I don't remember the URL now, but he had put in a list of all the top domains that had been sent. That it sent mail from. I don't know where it's linked from it. Is it in reports? I like that fancy thing going on the top. This goes back and forth. OK, clear stat. Gmail is the most popular domain in this country. I mean, look at the difference between Gmail and the next one. Now, there's something interesting that we found in this. Look at this one. This is the first non-webmail domain in this list. It does not exist. The most popular domain that is not a webmail domain is an invalid domain. Someone has clearly been spamming the regulator. So it uses our template. So someone used our template and spammed the regulator from an invalid email domain. We don't know who this is. Who did this? But that's an interesting side stat. Now, the point I want to make is we keep thinking about all of the fancy technology choices you can have to make scale and all these fancy diagrams about how many different moving parts exist in your architecture. And yet zero moving parts, and we still had scale. And the point I want to make is sometimes you don't have to try so hard. You can still do it. It's easy to do scale without bothering to do any kind of fancy technology at all. That's just the point I want to make. Thanks, guys. So if you have any questions on this, I'm happy to answer. Hey, so what percentage of the mails that TRI got came from your campaign? Do you have any insight from this? So the problem is that the TRI website is not stable. But since over the top already has the whole database. They have less than what we have. Sorry? They only have 800,000 emails. We have 1.05 million. So what's happened is that there's a lot of mail that TRI has not put out or they've not been able to put out. We don't know what it is. We don't know if it's someone did your thing or we don't know if them trying to revoke email IDs and not being able to do it or what. But we have only managed to get 800,000 emails out of the database. OK. So out of this 800,000, how percentage came from the same new campaign? So I don't have that report. I don't know if he's put the report out. Let's see what reports he's put. Nope. There's no report. So he's not put that out yet because I was asking for that and saying that can you please separate the ones that we sent and get us what else is left? OK. And I think he's not managed to do that yet. And now the deadlines for counter comments is passed. So now it's just an academic exercise. There's nothing. There's no value to doing that. Yeah. Good. Hi. So where was this actually hosted? Heroku. Did it cost for you? Heroku. It's free plan. OK. One dino. That's all it took. OK. Wouldn't putting it on a static website as in S3 would be cheaper? I'm sorry? No, so the thing is we wanted to keep our backup plan of doing fancy stuff. In fact, there's fancy stuff that you guys have not even seen. Like if you go to save the internet, there's an expert mode which Arvind built. Arvind is not here today. He'll be here tomorrow. Now what he did in this expert mode version is you could go through it like this and say here's a question which answer do you want? There are multiple answers. There's multiple choice. You can compose your answer to ensure it's not the same as the previous guy. So we did all of this stuff just because we were so kicked about doing something, you know? And the thing is this is a bunch of kicks saying, you know, I'm doing some typos in the evening. Let me go see what fancy stuff I can do. But the other people in the campaign, the guys who are the lawyers who wrote this thing said, boss, don't do this mix and match, you know? You don't know if the answer is legally valid or not. So we didn't go public with this. It's still on the site. It is just not linked from anywhere. So it's not been exposed. I heard the Gmail rate limit. Did you know it before? No, I didn't know the Gmail rate limit until we started doing this. OK. Because I've never hit the rate. No, I didn't. Before we started bouncing, did you know that it is going to bounce on? Yeah, so when we started receiving so much email, you know, somebody had this thought and said, does Gmail have any rate limits? And then we went and looked it up and said, oh, shit, there is one. Kiran, so Gmail, if I recall properly, Gmail mandates a phone number during registration for verification. Yeah, so you can have five accounts per phone number. OK. So your crowdsource phone numbers. So it doesn't mandate that you use one? No, it sends in passcode. OK. So your crowdsource the phone number and say, OK, boss, send me all the messages you get. So like stairs, for instance, sign up his mother's phone number, you know? So his mother has five Gmail accounts that she's not aware of anymore. All right, thanks, everyone.