 Hi, hello guys, small fellow. So this talk is going to be over, keep us updated. And all the links I've had are in the building list. So my name is Aarjavan.com. And I sort of single-handedly designed, developed, and made this site project with mine for about three and a half years, before shutting it down last year. And it's now completely open-sourced and is available as a regular self-hosted article. So about G-Past, so G-Past in its very simple form is a single-use right-to-back service. All that means is you have some products you want to try on any form or something in there. And you get a back-to-G-Past. And it basically sends you guys over doors to e-mails, push notifications, SMS, as in and out. As much real time as possible. That's what the service was all about. So the service was launched back in June 2014. And ever since then, as I said, a single-handedly doing the product management, development, design, social media management, customer support, all the roles that you have to play when you're running the company. So just to give you a brief idea about this game, I think the service was operating hard. Because this was not a company to say, but it was a sign-project, which I was working post-work. So it's not a huge scale, but a good enough scale for a single developer to sort of make it. So at the end of the study, now it was serving about 7,500 active users, tracking more than 40,000 products. And so the second was 6.5 lakh price-over-hours. It had been people saving up 15 lakhs over time. The company was confident in people's making money in the sample and we go from Amazon. Average commission coming out from Amazon with parts, not the non-inclusive type was they couldn't see as that happened in the system. And the product sort of made a lot of action when you want, you know, it made the product aren't half the usage data or half the user for five hours on the spot for a long time. React folks, each other, are part of the publications that sort of ordered this, the number one price tracking service for Indian e-commerce companies. And it also happened to me, we shared a lot of file, you know, folks with a lot of reach on Twitter that brought in a lot of organic reach back in the system. So I didn't really have to spend anyone paying or making an ad campaign to make the service popular. So a lot of people are saying a lot of good things about, you know, servers which brought in a lot of good things happened, like people, this kind of thing brought in a lot of good things. So before we jump on the board of the talk, a brief moment of time, you know, all those folks who work behind the scenes doing open source development, all open source technology is pretty much made this, you know, scientific possible. So all the learnings I had, you know, by integrating and retaining them, so I started talking to ways, one, how I was running when it was running for public versus how it is today. So these are the learnings which I had while I was building the servers and maintaining it myself. You know, you can have functions which will load it up in real time. So that keeps your start process pretty likely, right? And if the company had to start through loading on a deal, the process may kind of keep increasing because you're loading all those files, you don't have to process it up. So look at this function and the issue if this function is called, that happens, and what happens with learnings machines is that a learnings machine has an upper limit to the number of simultaneous files that can be opened, right? So how I discovered this problem was I was using this function to basically send a mass e-mail to all the customers on Chifar and I did a time and a number of users were less than 1024, this function just kept working. But as soon as the number of users in the system became 1025, my program stopped working all of a sudden, right? And learnings machine is a scientific answer process. It helps with a lot of this, right? So all you really need to do to fix this is move those language to the top of the stack modules and it just works. So you have to be really careful about the standard tutorials unless you're really sure about what you're doing. The DB schemas, right? I mean, we as front-end developers hardly get to touch the package and design schemas and the concept of a good DB schema or modern schema design was pretty alien to me back in the days. So if all I was trying to solve, I had to store those premises which I was creating into my game somehow, right? Especially, I had come up with that. So there is the, you know, containing the date and the price and what I did was attach this as an array within another document. I'm using MongoDB for instance, right? So it worked pretty well until the number of items or number of price histories were low. As soon as the number of price items kept increasing, the system started and getting slower. Why? Because insertions, applications, invitations with arrays don't really stay because the system has to sort of copy all that array into your memory then process and then what an operation you're trying to do with that. So what really helps is unlocking those array elements and storing that as an individual document within the creation, right? Instead of injecting an element inside an array element, and what happens because of this is MongoDB or any most equal database for that matter gives you a lot of arrays, linking, aggregating or doing operations in multiple documents. But it doesn't really expose many arrays for, you know, doing things with arrays as it has structures in general. So, memory is a big thing. But if you tend to use the arrays in which the MongoDB exposes, then it tends to keep the process system over pretty likely and it just ends to be ported. Now, when we do the same discussion, this was called insertion of data in the arrays. Let's look at the Shintas homepage which used to look like this. So, what this means is it shows all the products which anybody in the system is tracking sorted by some logic which is how many people are tracking all the nice histories that have happened with little people from high industry. So, maths had to go in with a little bit of data secrecy. And skills were basically used to find at any rate. Now, what don't find these Shintas is that it copies all results in memory, depending on what about where you give it. Grab all the results, copy back the memory and now we were maybe using Node.js to process that data. And back in the days when the number of products weren't even that much, it was taking up like 4 minutes to create that case on the home page. Process everything. And that is it just kept growing exponentially with growing number of products as well as product crisis trees. So, as a result, I had to create corporate data. So, I started out with using that that so instead of you copying the results in memory and then processing Node.js, you essentially use those documents which you just created in the collection. You can run your summations or whatever you want to do and then write down those results. So, you can basically write down those computed results back and then running everything and then you win your last two Node.js. So, that sort of brought down the time from 4 minutes to 8 seconds. You did an innovative process might sound like a good idea but as soon as you are dealing with a large dataset that it makes big improvements. It was all possible in the back end. For instance, if you have to take a, let's say you have to generate an invoice for a, so in the back end which will take a screenshot of your sub-HTML page, generate a PDF you will attach it in the email and send it to the customer. And that's how it generally works. And that's a pretty common use case of you know, spinning up a new process in the back end. I was doing something similar. I was basically using a phantom.js wrapper or capture.js which can accept any selected on an HTML page and basically they have a screenshot of that you know, normal event. You can't find the Amazon update with its com structure. It removed the selector I was trying to wait for and because the code that I had written it was really handling the case when the selector isn't available. And this Node.js process thought, okay, this the screen started to come out so let me try to spin up another process but the selector wasn't really there. So, you can imagine what happens. It's probably an 8G machine you have running a distribution and Node.js keeps spinning up new phantom.js processes right and you are out of memory in your time. Right, so this whatever communication happens between Node.js and phantom.js let's call it some kind of a communication which happens if you partly control that at least have the time out after which that process gets decayed or dilapidated. So, we can't put when you are actually not sure if this connection gets broken, what's going to happen next. Right, so as I said Node.js created like thousands of phantom.js processes and it basically is good at the whole server. So, she passed those relying on third party sources to allow expires it would create Amazon, Clip cards, not being any website, any number on page you could basically escape from the system. So, it was in this exercise that I learned a lot of things dealing with websites. And there are a few lessons that I learned a lot of things. So, if you are making a packet to packet request then that packet request client does not really send a cookie by default because it's not a browser grape so it doesn't really have a user it doesn't really send a key. So, packets can't flag that of some developer who has written the code for Amazon. He knows whether a cookie is coming or not. So, he can't graduate it's a board property and it cannot but what happens is with that response it sends your cookie back and now he moves back the key and the other request does a cookie chart and now you behave like a user. So, to all the systems which do not let you make a request directly you can basically make two requests and take around the systems. In the sense that if you are making 3PM, 3PM, 9PM every single day then it can flag your ideas for such patterns. So, in patterns that you are trying for instance to set up a cron job to make regular script requests packets can basically flag that so what you need to do is to change the user details pay for some proxy services so that you make believe that Amazon service that you are a user there is another thing about Unicor now what this means is if you are building a fraud which essentially clicks on all the things on the page to spread the wings and basically click on more links so what the Unicor does is it's a hidden link which is not very useful to a user but it's a key for a fraud or a fraudal link to click and if any click on that link happens then it flags you for a fraud activity because a user can't really see that link but this link actually got clicked so you have to avoid the Unicor to have to be careful not only the links which are visible on the page get clicked it generally lets you easily do that and pretty much just use before you actually click on links so such things have to be thought about before you write a simple script so I spent in thousands of dollars and paid proxies because I had to have a remote system for thousands of users who were trying to use the service I think the solution is actually cheap paid proxies don't really work and the companies who provide paid proxies are very expensive so but there's no other solution I mean if you want to spread a service as much as Amazon and grab prices every few minutes or every few hours so I thought I said to myself how can I be the cheapest and can people save money but I'm not saving money myself so what can you do with my systems that can help me save money by maintaining the service and that can lead to a lot of research in terms of what is the one that makes it possible to free viewers of services and how can I make the service free for myself at least and that lead to eventually into this whole buzzword of course technology which is not any service you are relying on somebody else's service to get things done so I took it as a challenge which gives a whole problem statement and it leads to that to solve the same problem so you need a couple of lambda functions one key value is stored which could be diamondy, high store any of the services one good server which can be put up on search.sh, zip now all the functions can be from open cloud functions in all the numbers but this is what I went ahead with I went ahead with Slamla this is actually few cloning a few repos writing a few accesses and you have your own price of engine so it is something this generally looks like so it is a user who tries to add products via say Chrome extension it keeps on on Chrome extension so this node server stores this in some products collection in Google Firestore so this is how a user actually adds a product in our database there is one lambda function which runs in this Chrome it is always running on Chrome and the task of this lambda is to open page and create HTML as a data nothing else there is another lambda item which does a lot more things by a HTTP to our module server it also runs in Chrome which I talked about this task is also to pass the HTML which our browser changed and grab any good data out of it so it takes a lot of it and send it back to the HTML server so another problem statement is that a lot of users have entered all the products they want to grab and what we want to do is to grab all those products as a list that's the idea so if we're talking this sort of query in some lambda it says it makes that query with Firestore, grab all the products it spins itself up several times and this works but if you start to take a limitation you can't really do it you can't install it with Firestore's client-side like when you saw a lambda you're sort of in users' positions they have so what you do is you run a long job on lambda there's a node server to talk to the Firestore and grab all the products and now the code Z now server does is it spins one lambda function per product in parallel now this is the beauty of the whole architecture now what happens is every lambda invocation happens on a many ideas it is something immense you are essentially grabbing all the products which could be millions in numbers and you can spin up millions of lambda functions in parallel and you can still have a response path in under 5 seconds which was the 9th way to manage if I was managing my own servers at that scale so now the scale you're supporting is the scale of what your lambda is doing so each of these lambda invocations then creates those browser invocations which sort of you know, returns that way which parses this HTML path just on back to the node server which talks to Firestore and stores that new pricing and this now node server has new knowledge of the new price it figures out if a new price drop or invocation should be sent back to the user or not, if it is then it sends that back so this is how I sort of decoupled all of my services and made these individual services which sort of managed themselves and all you really need to do is clone a public network of reports provided in your keys and deploy them in subcabera it's great that it's cleared and ready you can support millions of components request out of the box, there's more maintaining browser instances there are automated IPs thanks to our Cloud Functions dispute and all third party functions work or fail in their own box this is a very crucial part as well because when you're doing third party functions you don't know what can fail there are like n number of situations which arose at a single line and I had to put my ways on what's happening, what went wrong so when you think about what's the responsibility in specific machines you know exactly which machine is paying and what task what did we work on right so I moved from self-hosted 9.js or video sort of installation to homes on lander I moved from a self-hosted 9.js installation to file storage moved away from paid talksies which was a huge saving moved away from servers and load balancers to emcee now all three services all have amazing three tiers so you are pretty much running a service if you don't complain to social servers like I was doing so I'll share with our clients to use their post installation alright so the next part of the discussion would be about all the product work once I had sort of creating this as an individual owner so what I learned is if you create your backend flexible enough to be able to I created a bookmark link there were pro extension, file extension an iOS app and write up another website like there were seven clients it was possible to simply add a new product to the service and as you push notifications browser notifications so if you make your backend flexible enough then when touching your backend it just keeps spinning up more more clients there was this one thing before of building an iOS app I had to design this I had to create my own components and when it happened I could just reuse the same react with components which I had coded for iOS because I was an instinct but the design would be just 100% used as it is if you're kind of working on your own side project so this is how the iOS Android app you see that viewer card coming in and it doesn't appear on the app in sports just kind of all these price drops which are happening sometimes because we want to Twitter all the time which led to Google believing that it's a genuine website we will actually do a video about it on this app, number one Google is searching for price orders she passes the product survey if you and I have seen we'll have separate price streams for that and see separate graphs for it so nobody could really see who is stuck in the system so there is no possible they delete a product in the system you see in the drawing in the system what it becomes in the embryo we would have removed it other indications and what not it can be used from Amazon with the same captcha so what it essentially did was right before sending the email it would propose a date page take a screenshot of that inject it so anybody can see in any way going on one of these sites and that led to amazing openings for the Indians because people thought that even if they are not interested they are at least there to see a date which has just come up in the email box without any opening Amazon would trip out so this is how it used to look like somebody cutting the third board display price drop happened but a real time we went in and I say it ends in 19 hours so it sort of is a way to open people I used to use this service for postcard average here when we were starting this project we talked about this famous and what it was I basically partnered with FreeCharge to earn their view and I tell them I will promote you in my emails and I told people if you read about me on social media they are going to use some free google and they actually did people actually started posting pictures on facebook about trying sushi bars just for some free google so people do that so that's pretty much it for the talk I started this to solve my own issues I wanted to buy a phone and an iphone and whenever I would so that's what I was thinking I thought I would buy a phone but the next time I do it I really want to automate this process and I said please and that's where the simple input box of the page which didn't have any other input just send me so people started to at least let me input my email so what if the customer is hard to say I listen to them and let them tell me what to do with this product for them so there's a couple of products to be asked it's okay to have outputted IDs I don't know at the time of shutting it was shutting it down it was still running on the work point to download and it's okay to not build anything in the first scope that you are giving off because often we hold ourselves back to build something because we think here let's not put it out let's not put it out because that's not the perfect thing that I imagine and it works really well that's why it was so important it's great to get feedback and listen to customers because they know exactly what they want and let them tell you what to do if you are at least building a site project or something to do with it it was shutting down and pretty much we don't need to be doing that all we would really do was see a list of products and just receive push notifications only when people said that they wanted to ship and you will never really finish that site project of yours otherwise it's just basically brilliant