 Hello, good evening all Sorry, we got late actually our taxi or driver went to some other place some misunderstanding. So we got late so Today the agenda is Firstly that I will present when deploying my end on AWS with S3FS and Second there was some presentations by AWS solution architect, but somehow we couldn't make it happen So today we have only one presentation here Before we start I want to share something about the magento meetup. We started around four months ago. So the idea is to increase the magento community in the Singapore and The magento developers can communicate each other and can share their problems can share the knowledge how we can do the things on magento and depending on development also and server related things hosting great things cloud Etc. So this month we proposed to have on the AWS centric one Also in the last to last month being organized in the Microsoft Office this month We are going to organize AWS. So we thought that it's a good idea and as I'm working in Renault's technology And we also worked a lot in AWS. So We want to share something how we utilize the AWS So let's start it Okay, I should not go to the full screen Let's do in the presentation directly. I should continue here. I support Okay, so The agenda of this why AWS? I will firstly brief about the AWS also why we should use AWS and the second thing is the auto scalability how you can make Your architecture is scalable so that can handle unlimited amount of traffic and you can How you can organize the resources in the AWS architecture? So I will share with that So firstly you may be already aware that AWS is very big it's around 45 plus services already in the market and H per your need you can utilize the services and AWS is very very Flexible you can just use the services which you want the services you do not want just do not use it So you just need if server you just use easy to service if you are need a story You just use S3 even you can use your other hosting providers and Combined with that you can use the S3 storage only so every different services is Flexible and independently isolated from each other. So you can utilize the power of services And if we talk about general purpose application web application Including Magento, you do not need to use so many only four five is enough to use so let's briefly see What services we can utilize and a general web application system first is easy to issue is elastic compute cloud. It's As you can say is operating system here You can run any operating system and it's virtual machine on the cloud Second is RDS RDS is a relational database service All you know that AWS is infrastructure as a service, but on top of that RDS Because in infrastructure service you have to manage everything at your end, but RDS database They are providing the managed service. So within the RDS it inbuilt have backup system and or update system so RDS is most widely used whenever we use the Application and RDS provides some database like MySQL, MSSQL, Oracle and PostgreSQL and Aurora also it's their database. So these kind of database you can use and can utilize Next service we want is S3 simple storage service to store the user content and Keep the user content outside of your server So you can have a scalability at your instance levels the application layer. You can have this capacity and keeping S3 outside it's safe and also on top of that you can use cloud front CDN service to to deliver your content and Fastly from nearby location of user location Second next service you can utilize a root 53 root 53 is a managed DNS service by AWS So here you can add your domain create hosted John and Point your EC2 instances or your Lord Valensor to point your site and Next cloud front. I already talk that it's a CDN service from AWS. You can fly Cloud watch cloud watch next service you can realize to monitor closely on your instances like EC2 RDS both have instances They provide free monitoring of five-minute interval Monitoring you say it is CPU utilization Network input output you can monitor those at five minute interval freely, but if you use cloud watch extended Facility there is a facility to utilize at one minute interval at some extra cost. So Depending upon that you can take some actions Like you say that if CPU utilization is higher than 80% for such amount of time take this action Like you can start new stance. You can alert send the email SMS those kind of thing you can do with the cloud watch monitoring So that's very helpful for the network administrator part Next service you can use the elastic load balancing elastic load balancing You have to use if you want multiple instances of your application That can handle the load So I will share more about this letter Elastic finish top. Yeah, so we see is infrastructure as a service But elastic finish top is you can say the platform as a service. They are providing So in this it do the provisioning of resources you can set up the resources type and it will start the resources for you monitor those and You can you can do everything like you can do as a platform as a service and You can utilize your custom AMI also even in that case, but it's a mixed model actually So it is not me. They are providing us There is many AMI you can select but if you want to change those AMIs you can just aid your custom AMI where you can set up your Operating system like linux and set up of your tools like open-office or ways, whatever you want to run So you can realize that So it works with elastic load balancer closely and easy to S3 also So these are the most important services which you can see in a general web application system Yeah, there are many more we can utilize like there is for for storage only we have the glacier for Like for the lambda is there for the mobile testing is there Like mobile farm, I suppose there are many other services we can see now I Want to share about how to think a scalable point of view how to think from a scalability point of view Starting from single box like you see only one easy to instance We have and everything is in single box like you have the always there you have your apache there You have my skill inside there. You are storing your user content there. So everything is single box With AWS you can use this like and it like similar to any shared platform or other platform So we start from this the user is directly hitting your server This diagram is actually expert at abuse architect. So regions AWS is divided into the regions. That's their main data centers Within the region there can be multiple availability zones that are connected using lands or some other thing Probably they share around five to ten miles distance with an region. So around I think eight regions We have including Singapore, North Virginia of California, Sydney and others So see firstly, this is a single box now think from a scalability you cannot a scale it so Adding the services I will add the more services like first. I added the idea service Keep out your database to RDS if you're using my school Postgres oracle you just keep out into RDS set up RDS import there and connect your application to that RDS directly There's a lot security also implemented. So you can set security group rules like only Only you're easy to can access to the database because you do not want to access your database from other side If you're working in a company you want to access ideas from your company network You just set up the IP address in the security group of ideas and only your company network can be able to access Your ideas those kind of security you can see Now think more you can add the S3 bucket For your user content your application is one but after that if it launched user edit like profile They are uploading profile images you have product product image So you can store those into S3 bucket so Then your load will be also less like your static content Even jcss static content you can offload to S3 Later if you want more a scalability you can add You can add a load balancer there and with the auto scaling policy Auto scaling policy you can set up as per your Application nature like you you you want like if your application is running your easy to instance is running 70% or more load for Less or more than 10 minutes. It's not another easy to instance So because you can start now now you can start many other instances because it is not depending upon anything It's just a pure application now database is outside Storage is outside user content outside. So you can scale here at this level So you have to start a load balancer. So the load balancer you are will be connected to your DNS So wherever whenever someone will access your domain, it will point to the Load balancer first and load balancer will see to which server the service that request should be served So you can see this way you can make your application more scalable If you want more scalability, you can do your deployment to availability zones so if even Due to natural disaster if one ability zone is down or some hardware failure one availability zone is down It should be there is one more availability zone where your servers is running and you will never let your application down in this case you can also have master and slave database one database you can make master and Secondly the slave database you can use redeplica one set pal. So it's up to you. How much scalability how much? for tolerance disaster recovery You want it's just a start. So you can see Adding more services to this you can use on the S3 bucket. I say you can use the cloud front For CDN for fast delivery of content You can use other services like cloud watch. I am role. I am role to manage finally on your resources if you are working in a group and Want to give access somebody for S3 only for easy to only you can control those HCS service you can use for its email service. You just get a SMTP utilize that and start sending emails Just like the sand grid you generally use or gmail as a SMTP huge so this this is the planning or You can do to make your application is scalable So this was general public application this applies to magento also this applies to general public application also if we talk about specific to magento, you may be all know about the magento it's a e-commerce a big e-commerce platform and many sites are hosted on that and running and In magento, we have a media holder where all the product images resides If there is users profile images inside there So only the media folder is the one which is user content after the application application in magento You can host your application in the easy to instance, but media is tied with that. So media need to be there So either you can use S3, but you have to do some programming coding To make whenever user is uploading in the back end some product images. It should go to the S3 There are a couple of ways we can discuss But today I will focus only on using the S3 FS and I will tell you why because if you do some coding efforts it will take time and If you'd like S3 FS, so let me share what is S3 FS. S3 FS is huge based file system Huge it means file system in user space It's a technology which allow any user to create its own file system without going into kernel detail So you don't need to modify kernel. You have a file system. So You can just create a folder mount any drive Generally, it supports like S3 Google Drive also. So This is a huge base file system where There is a library S3 FS and in that using that library you can mount a S3 bucket to your local folder So what we are going to do is we will mount our bucket to the media folder If you mount the media folder, it means whatever is in the bucket is in the media or whatever we upload in the media folder It automatically goes to the bucket Yeah Since S3 FS is a huge base test But let's say if my Compute mode is Windows Okay, would it still work? No S3 FS is only for Linux Only for Linux? So no Windows support? No Windows support for S3 FS library. Yeah, maybe some other library support for Windows But it's only for Linux So it just like mounting like you see some power ISO tool in the windows you see power ISO So you can mount your ISO into a virtual CD drive. You don't have CD drive, but it's still you mount It's having this e-drive or f-drive similar to that S3 is outside, but you mount here So your operating system assume that this is here. So whatever data we have in media. It will assume the data is here And that's why you don't need to change anything in the margin to recording because Whatever product images you will upload it will go to the local folder as per programming But immediately it will sync to the S3 bucket So why we use S3 FS because you can use for media folder and it's scalable because keeping outside the media folder Will have a then you will be able to scale your EC to instance like then you can start to see to instance You can have a lot well answer because both servers are running But the media both is Using mount concept of S3 FS and mounted to the same bucket So sharing the same data Storage so either you are connected to first instance or second instance You will never know which instance you connected But either instance you connected if you are uploading something user content It will going to the same bucket and immediately sharing to this whole EC to instances that way you can achieve some sort of a scalability and The main main point of this integration is no much into coding required Nothing change and it will instead for work So I can quickly show you a demo how you can do this I set up it You see I created a setup this one I just started an EC to instance and I set up much into there and In the media folder I mounted a bucket So I will show like where I am Here in the HTML Okay, let's see here. Yeah, you see This media folder you can see here, but actually it's mounted to bucket. I can show you how Let me show you firstly No, once you mounted is linked automatically It's automatically let me show you the first the demo. So later. I will mount or unmount also. We'll show you You see I have file here is screen shot and this bucket. I'm mounted here So you can see these files are this is a Reynolds demo bucket. I mounted into the media folder here So catalog folder customer folder. I already have here So I can delete one file here like I if I delete this screenshot from here Okay, doing shots showing here now remove Or even if you delete some file here like a skin shot if you want to delete here It will look from here. This is skin shot will be removed. You see this is skin shot is from here so You see this is the media folder here. You have the catalog product image edge So whatever image will go here will automatically sync to the S3 bucket and this way you can have multiple Majento instances and How I do it You see this is on the git already as 3fs and it's very easy to install actually If you go here if you started the imagine default machine image imagine machine image It's based on Santa s so you can use the power of yum yum install So you just have to install some prerequisites Which we need for s3fh to work and later you can clone s3fs from git repository directly And there is a couple of commands you can see to make it install and once it install you will need to set up your password file So generally I prefer we can have the password file the past wt file at some location and in that you have to give your Account access key and secret key You can create this access key secret key by I am roll you go to the AWS console And here you can create your user role and have your secret credential keys, which I already set up and Just it's there and there is one Commands which you can realize Let me see The command I noted Yeah So this is a simple command if you want to mount a bucket you just utilize s3fs and They show allow other renaissance demo is the bucket name at the folder name which where you want to mount Okay, and actually I already mounted so I will unmount it first So for unmount there is a command of a future unmount a future mount dash you Let's add this command Okay, so I have to log in edge Root firstly, then only it will work Yeah, it's unmounted So if you see this was the media folder if I refresh it now, there is nothing Because when we mounted it then only it will show so now it is not sync and if you want to Mount this media folder then you have to Use the command which I already shared like this one You come on copy and Just it's mounted and automatically your data will sync here Your voice will assume that your data is here and it will work like the normal much into works So it's a magic kind of thing you can see but it's actually the file system you're mounting So this is the concept. Yeah, definitely you need installation and some other but all helps are there So I'm not including that in the demo, but you can go here You can find all the details how to install s3fs how to set up your configuration there are a couple of ways you can set up the configuration and Then just mount and mount your bucket and enjoy So this is the demo and just Some troubleshooting may be because if you do it Your folder should be blank and your folder should be writable. I prefer 777 permission or appropriate writable permission Yeah, you can say the permission is not right. Yeah, the appropriate writable permission may need only but I just go with Yeah, we have a discussion in the last meetup also about this permission that The permission should be only writable or you can make the owner to Apache Then it's safe to use So this way you can utilize s3fs. This is the one way of doing the s3 integration Yeah, there are surely the couple of ways like you have s3 SDK by AWS officially you can realize that do some custom coding to handle your user content to store in the bucket and Do some path changes whenever you are referring the images and refer those images from the bucket so this is all about the Majento and firstly we talked about Scalability point of view how you think this is a start point you think start thinking How much they are a high availability you want for your application according to your client requirement or your project requirement So and later we discussed the one way of doing the Majento s3 integration. So that's all That's all from my side. Yeah, any queries. I talked a lot Yes, sure. Once you mounted your s3 bucket. Yes, does it act like a normal Linux folder? Yes, like a normal folder so we can just run sage mode Yes, you can create those settings. Yeah, you can get files folders inside that create give permission Like you do Linux permission you change the owner everything you can do But it's actually a file system s3 file system you mounted here So this is a concept of huge huge means file system in users. It's a technology which allow you to Launch or install a file system not natively supported in the OS So s3 is not supported in natively in the line expert. You can mount it here So is there any configuration on the s3 part? No, nothing You just need your access key and secret key to set up in the s3fs configuration, which I showed that by swd file you to create So access key column secret key you put it and then your s3fs may be able to communicate with your bucket So if you will mount it will show you if your configuration is wrong Your condition is wrong. Do you usually create another user for it? Basically, yeah, you can use the root user, but nobody Prefer that you create another role. I am role and give the only the s3 specific permission s3 all permission So that role if somebody can hack your credential history He have access only to s3 not your other resources of AWS So if you can use the power of that So here like the media folder yes, the s3 bucket is like Because we need it because I showed you Yeah, you can do that Yeah, you can do like whenever you upload your image you do your custom coding Integrate s3 s3 hdk of 4 php and then upload it to the bucket and where you are referring in the front end You change the path you use some method and refer the path to the s3 bucket you can do that also But my point was you have to do the efforts here. It's very easy to do without doing changes in your coding You also have the saving measures No, no, actually it is not taking any space on the on the client side on the OS It will not take any single space if you see the media folder size. It will not show any size Let me show you if you see media folder. It's zero size It will not take any single space here. It is just a reference here Not storage We have a No, no, no, it's not Internally even it's not downloading the files it just give the reference So if you access it then it download and cash it on the local OS So if you want to ask again, it is in the local OS So it's fast, but whenever you access first time it create the cash So if you're excited you mounted and you launch your site So whenever user is accessing your product images, it's downloading to the OS Or no matter which OS because you have multiple OS you can have Like using this you see you can have multiple OS easy to instances So it will download the image each user will access them Definitely for WordPress also you can use Drupal also you can use As long as you are having one single folder of user content If you have multiple user content folder, you can utilize multiple buckets You can have multiple buckets mount them multiple times. You can use that kind of time Yeah What's the minimum easy to instance type Do you think if I just want to have my Lajento in one easy to instance What type instance would be the minimum that it could handle that Lajento, because it's like You want to do and the production or just for your development production production. Yeah, production At least you need a small instance 1.2 gigabyte trim I think for a small instance. We have the nano instance. I started this instance nano instance It's only having 600 megabyte RAM Yeah, it won't for production. We should not use that because whenever you will access 20 or 50 concurrent users Majento takes a lot of RAM you already know So every request takes a lot of RAM probably like a T2 video T2 medium is very fine. It's fine. Yeah, very good. Very good. It's 4 gigabyte RAM medium instance Very good micro Micro nano will not work. Yeah from a small medium or bigger you use a small snack It's small is for low traffic. It's enough. Okay, even for our clients. We are using C3 dot 2s large 64 gigabyte RAM server. Oh, that's huge. He's huge, but their traffic is huge So we we are using for one of our client is Charles and Keith and for them We are using two instance of 64 gigabyte RAM So to production instance and then your database is all of these other ideas in the ideas isn't different That is also running 122 gigabyte RAM server Because ideas here ideas is also have its own Instance yeah, we don't know about the OS layer about that But it's an instance where you can control the instance type the RAM You can control so but ideas the minimum instance will be fine Ideas minimum instance easy start from micro. Yeah, it's also start from micro small is similar similar instance of EC2 So if you are using medium for the EC2 you should use medium for the ideas So you can do this kind of auto scaling that time because if you do you start with a small instance Even your traffic row Grows you can set up auto scaling policy in that you set up like when the load is More than 70% for more than 10 minutes Start another instance. That's all the load balance Yeah, or the load bansal level you can do also you can shut down instance like when the load is less than 30% for more than 10 minutes down the one instance because your load bansal is handling the load So which server is active it will send the load if server is down it will not send the load You can have your setting like how many minimum instance you have at least one You can have minimum through running all the time some client requires So like that you can have auto scaling and you can save the cost also You don't need to be provision all the time the high capacity which you are anticipating you can have a moderate capacity all the time and run It's for traffic So it's it's the main power of AWS. It's very flexible in this case any other cushion Be detailed ocean, it's not Claudia We have many other application on the dedicated or shared hosting or Anyone on the Azure cloud if you talk about cloud we host an AWS Azure Google App Engine and If client budget is not so high we can use the shared hosting or dedicated hosting Okay, anything else? If you want to share something, it's it's your platform. You can share your knowledge or you want to share some announcement Thank you and thanks AWS for