 Yeah, I'm stuck but yeah, I don't find my mouse if someone sees it, so the computer is not the real one. There we go, marvelous, I will move away now. Thank you Daniel. I think the room is a little large and these walls absorb sound, so please do keep the volume as you go. Okay, so my name is Daniel, I work for a company called Spool. It's a streaming company, OTT company, and we stream Bollywood content around the world. So obviously, to stream content around the world, you're like a big user of CloudFront. That's actually our main cost in AWS is CloudFront. Okay, so yeah, so we deliver video on any kind of device, mostly on Android and iOS devices, tablet phones, also on laptop, desktop and connected TVs. So as I said, it's really like it's not targeted to one country, it's targeted worldwide, which makes the problem a bit harder. So especially when you have people in this region here that have like tiny, tiny internet connections, it becomes a tough job. Okay, so to send that to everyone, we decided to use only HLS, so to focus on only one format instead of starting to try out different formats depending on platform. Because when we started Spool, we were like four or five tech guys, and there was one company in the world that said to us, you have to use HLS. So we stick with that for other devices too. So for those who don't really know HLS, it's kind of an interesting format. You have like a master file that describes your content in which you can put different renditions, which each rendition has a different quality, be it on screen size, on bandwidth and compression. What is interesting to us is mostly the bandwidth. So the devices will automatically switch from one rendition to another depending on what bandwidth it's available to them. So it means that normally, if everything works right and well, the user will just get the best quality he can get with his actual internet connection. But that's the theory. So everyone of this rendition has then a list of segments, which are like tiny, chopped movie segments, generally about 10 seconds. And it's just pointing to everyone of these tier segments, which is a part of the movie. So that's what the user explains. So the player will try to fetch the 4000K content if he can't, because he's down speed is only 980. He will switch down to the 164K rendition and get something more grainy, more pixelated, but the movie should never buffer actually. So the first question is, where do you want to store and how do you want to serve this kind of content? So the storage, if you work with Amazon, I think it's pretty obvious. It will be S3. Then how to deliver that? First question, should you deliver directly out of S3? I think most people nowadays know that it's really not a good idea. First, S3 is really slow. And second, it's S3 is only in one point in the world. So if you have US3 bucket in the US and you want to get a file to Singapore, you have to download from US to Singapore. And all of you that try to open websites in the US on a Sunday evening, you know how painful that could be. So the real stuff that I think anyone does is use CloudFont to serve that with the file directly into S3. But CloudFont is still not good enough. So yeah, I should have put it into Y, but how come? So CloudFont not good enough, how, why? So there are a few stuff that, like, as an end user, you look at CloudFont, you send it to, why don't they do that? Why don't they allow me to change this feature or these parameters? So first, you run into this problem if you have something like HLS. It's a lot of tiny, tiny files. So one movie is about 4,000 files that range between a few hundred K a file to six, seven megs a file. So it means that since we, if you have a lot of renditions, you mean that on every edge, a file will not be hit enough in one day. Because some user will hit a good quality, some other user a lower quality and so on. So every file doesn't get enough hits to become relevant for the CloudFont algorithm. So at the end of the day, CloudFont will say, oh, this file is only 100 K and was hit 200,000 times. It's not worth it. Throw it away and leave this huge MP4 file that has 20,000 hits, but it is a 2GIG file. Leave it in CloudFont. Leave it long in CloudFont and the small files just throw it away to make room for something else. So with that we realized that actually the TTL we were doing for CloudFont, we went like just every file is like a one year TTL. Keep it in the cache for one year. That never worked after one day or even a few hours. Generally the files were gone from CloudFont. And once the file is normal in CloudFont, your big issue is that CloudFont will fetch again from S3. So if I take back my example before your bucket is in US East Coast. You are in Singapore. You obviously hit the Singapore edge. The edge will fetch from Virginia. So actually your bottleneck is no more your phone or your laptop connection. It's the connection between Singapore CloudFont to your Virginia edge. And this one is slow regardless of how and you can do nothing about it because it's not in your hands. So it means that every user will get these slow files. And one thing that happened for us a few years back now it starts to get a little bit better is you cannot rewrite headers. So for those of you who have worked with Vanish, Vanish has these four components like request in, request to backend, back from backend, back to users. On every part in Vanish you are able to rewrite anything you want. In CloudFont you have zero control. So you could actually not say like remove this header because it just adds a mess. So that's actually my next example is CloudFont will store differently if your client has the request accepting coding jzip. You will just say oh it's a compressed content. It's not the same answer I should give as a non-compressed. So you will store that in two different storage locations on CloudFont. Even if the end file is never compressed on a movie segment you never compress it. Because it's already compressed due to your MP4, H264 and all that. But it means that a browser that will by default add this header and the phone that doesn't have it, a web app, a phone app. They will start to hit the same CloudFont edge but use two different buckets on the edge, two different storage locations. So they're not even able to share the files that are already in CloudFont. So you will start to have double storage and your movie will be fast with a phone but slow with a browser. One thing is hard, you cannot warm up CloudFont. So you have your 4,000 little files on S3. You put your movie live on your website. CloudFont is empty. So it means the first user or the first few users that come to your movie, they suffer and then you generally get a bad tweet about it. So yeah, because the first user stretches from S3 obviously then... And you cannot anticipate the next request. So on an HLS or even on an image gallery, if the user takes file A, you know that in 10 seconds he will want to have file B. And if you have a HLS or you have anything that's like sequential, you know the process. You should be able to tell CloudFont A, this guy is pulling this file, please start pulling like the 10 next files already. And so yeah. And generally the S3 buckets are far away from your CloudFont Edge. You have only one bucket that you generally put in US East because three years ago it was the cheapest, like way cheaper in Singapore now it's no more the case. And now you have bucket replication that help a bit to come to that later. And so all this problem what happens is since your player, you have to remember that since your player is choosing the rendition he has to take, he will get to one, he will be able to download it because it's in CloudFont. He will say oh, I download this segment really fast. I can ramp up one better. But as soon as he ramps up to one better, it's no more in CloudFont. So it will be really slow. So oh, I have a bad network. I go down again. And the next down is perhaps not in CloudFont A either. And then it will go down again. And you end up finally with a really crappy video quality. Even if people are telling you on the support channels, I have a 10 Mbps line with my ISP. Why cannot I have a really good movie? It's just because how the player works. It's just like every segment looks at the speed you are getting it and then decides to go like one higher, one lower. And if you're always fetching from US, then for sure it won't. You will never have something great. So a few things of all these problems. Some things, some points you cannot fix. Some points you can work around. That's actually the magic of Amazon. You don't have access to everything. But you can build stuff around to try to make it better. The first thing we did was to warm up CloudFont. So what I did is just for every movie, I know the URLs because I encoded them. So I know what name I gave them. So for every movie, I create a list of all the movie segments. I found out somehow all the DNS names of every edge of CloudFont. And I'm just calling one by one. Curl, curl, curl, curl, curl. And that's just like a fleet of 20, 30 spot instance in every region that run with an SQS queue once it's finished. Spot instance down. It's a few hundred bucks to do that. If it makes like 100 customers happy, it's worth it. I tried to start to do that in London instead of having my spot instances. Yeah. Yeah. London, not that great for that kind of job. So I run into the timing issues of London. Like more than one minute it fails. Because I have really like a movie is 4,000 files. You have 50 edge, so you make the map. It's really a lot. I know that now we're still running with Lambda. I just make a lot of small bunch of batch files running only in Virginia. I would need to start to move that like more around the globe once I can have Lambda in more places. So then the interesting part is that if the S3 bucket is too far away, it's to use something like what we call CDN for CDN. So actually we put the CDN behind the CloudFont CDN. So we started to want to replicate that to every bucket in every region in the world. And then we realized it's on the cost level, it's too much. On the maintenance level, it's too much. Because we need to be sure that every file is everywhere. If one file is missing, the whole region, the movie doesn't work for one 10 second segment that's not there. And the other problem is that CloudFont can only access one domain. So you cannot tell the CloudFont, oh, if you're in US East, access this bucket with this domain name. If you're in this region, access this bucket with this domain name. So you can access only one domain endpoint. And two buckets cannot have the same domain name. Because the bucket names are unique. So what we ended to do is we fired up EC2 instance in every region, put on them NGNX and use the caching mechanism of NGNX to actually fetch objects from S3, put them in NGNX in the cache, and then have CloudFont pull out of this caching service. So for that you would need like a custom made NGNX build that is able to talk to a private S3 bucket. So if you don't want to make your bucket public, you have to add some key headers to access the private files. So there's someone who made a plugin called AWS Oath for NGNX that allows you to do that like easily. Access key and secret into the config and giving the bucket names and the path you want to access to. I added to some lower capabilities. Actually I used lower because it was already there. And then at the end, the user will still go to CloudFont. But instead of going directly to the S3 bucket, we use a Route 53 latency record to point every CloudFont edge to the nearest EC2 instance, which caches the content and this EC2 instance will fetch from the S3 bucket. Like that. So it means that instead of having to fetch your file that is not in CloudFont from your US bucket, you will actually be able to fetch it from your fake S3 bucket that is an EC2 instance inside Singapore, which already you win quite a few seconds on that. So the benefit of using NGNX cache instead of full replication is you don't have to manage the full replication. It's just a cache so it gets replicated when needed. If some files are never requested, you don't waste storage on them. And since there is an EC2 instance, you have an operating system behind and with an operating system, even if managing server sucks like you said, you can do stuff. You can toy around. So one issue with the NGNX cache is an EBS instance. Yeah, an EBS is only one terabyte. I didn't want to start to manage multiple caches with NGNX. So I just went the easy way. I spin up an LVM and I start with a one terabyte. Once a one terabyte reaches 80%, there's a cron job that just checks that and adds another terabyte and expands the LVM instance at the end and then I put a cap to manage cost. So now I think I am around like six terabytes maximum of some instance. And if the cache shrinks, it doesn't shrink because you cannot hot shrink an LVM partition. There's still one issue with all that, even if it works fine. If you want some redundancy, you would have multiple instances in each region. The problem is since it's an EBS volume, you cannot share them between different instances. So if you have three instances, for instance, that run in one region, they will have all their own cache. And with the random ELB connections, you won't know which one has it. Perhaps sometimes it's in cache, sometimes it's not, which is a bit not optimal. But I hope that Elastic File System will solve that. So with Alex, we need the Elastic File System quickly. One, two, first. Okay, so one other problem I had. So with that, I wound up my, I'm able to wound up my cloud front. I'm able to move the content nearer to the cloud front edge to anticipate the next request. So you know if you take the 1,928K first segment, there's a really high chance that in 10 seconds your client will request the second segment. Or perhaps one higher, or perhaps one lower, but number two. So what we do is when engineering sees a request for segment one, we actually start to download already before the client requested segment two of the same rendition, one up and one down. Here just one up and the same. And then same for once segment two comes in, when the client requests for segment two, we start to download the next one and so on. So it means that as soon as the client gets to segment two, he doesn't have to fetch it from the end, or the end of the world. He can fetch it from something that's almost on the CDN. Let me go first. Is that the right one? This one. So I started first to just say to NGNX, like launch me a system command to download the other one and I realized that that's actually a blocking thing. So the client has to wait that to get segment one, he has to wait at segment one and two at the end of it, which doesn't really make sense. So I use something, I use the queue system to have it non-blocking. That's where the Lua script comes in. So when the first request comes in, I just send with a Lua script, I call a system command that allows me to just throw the current URL into a Beanstalk D host that runs on localhost. Then I have a Beanstalk CD-man that just reads every URL, split by the slashes, finds out which segment it is, build the new URLs with the next segment, one rendition up, one rendition down, and starts to do curls on that, which actually not curl in Python, just downloads all the segments using localhost. So if I did that on localhost and not using SQS, it was just for matter of speed. Like I didn't want to have network calls going to outside of my instance waiting and then another process that was then calls again. Having everything on localhost was just to improve the speed of the whole process. And that's the part of things you can, the cool stuff you can do once you have access to an OS and be able to toy around with it. Okay, so I didn't dare to make a live demo. I was a bit busy today to build everything up. But I did some tests last week, just to have some number in case I wasn't able to do the demo. So I had a bucket in Virginia. The client was not in the AWS office at my home, but the bucket, the cloud front end was still Singapore. And I had an EC2 instance in Singapore. So if you see on the left one, it's just cloud front directly to S3. So the transfer speeds for every segment, which are like four to six MBs, it's like in a range of four to five seconds. So with that, it's still okay. Because remember, one segment is 10 seconds. So as long as the client has less than 10 seconds to download a segment, you are good. So sadly, I have a too good internet line at home to make it work for real. But once it happens, and I have seen that even in Singapore, it's that suddenly the segment takes 20 seconds to download. At this point, the player will start to scale down and take something that's less good in quality. But the interesting part is once I use this mid-tier servers, the first one is still the same speed. There's no change because the file is not in cloud front, or not in my mid-tier server. And not in cloud front, obviously. And then the second one is directly fetching from this mid-tier server and no more from my S3 bucket. And you see the jump in speed is, I think numbers speak by themselves here. While it's lucky we're in Singapore, Singapore has a good network, servers are in Singapore. So we're all a bit cheating in Singapore. What is the precepts of this, the fetching from your mid-tier to cloud front and then the download from cloud front. So that's the download speed from cloud front. That's the download speed from my computer at home. Yeah, from cloud front. I made sure cloud front was empty. So once you have cloud front, it's even like 0.2 here, something like that. What I wanted to solve is the worst case scenario, the files are not in cloud front. So that's what I'm trying to solve here. I was interested to know what the speed was from cloud front, from your mid-tier to cloud front. Yeah, it depends on which side you look. That really depends. So in Singapore I think it should be like gigabyte or 10 gigs because it's in the same building. But if you look, a customer that will be in Mumbai, he will go to the Mumbai Edge hopefully, and the Mumbai Edge, his nearest EC2 instance for the Mumbai Edge is Singapore. So then it's actually in these scenarios that this stuff is interesting. In Singapore, everything works. You could pull out directly from S3, you have no issue in Singapore. So that speed 100 is a limit from your home connection? No, I think that's the limit of my EC2 instance. So you mean bandwidth of... It's M4 large. So it means your outbound bandwidth limit is 1.38 or 1.38 Mbps for your EC2, do you mean to say? Yeah, so here it's at 100 Mbps here because I think the bottleneck is the network... Let's call it a network card, even if it's not a network card. It's the network interface on my EC2 instance. Oh, EC2 was gigabit. And just how fast is your home connection? Like 1 gig, like every Singaporean, no? Has someone an internet speed lower than 1 gig in here? It's a tech room, no? Like everyone is with... I got 2 gigabits. Yeah, I wanted to go to this one, but I had a wife at home then. What for? So yeah... It's free to catch me afterwards and we can talk about EC2 instance networking as well. I want to hear that too. Okay, I guess I'm doing a talk on that then. The question is actually what he's asking is... I'm also interested in the same area. So your test is with mid-tier, without mid-tier, right? Yeah, so... Two testing that you want. In both cases, there's CloudFont. Yeah, only the mid-tier is not there. Yes. So one is actually picking... CloudFront picking from S3. Another one is actually is picking from... using the mid-tier to pick from S3. Yes. Right? So, but if you notice that the either case, as an end user, I'm using my home network connection. Yeah. So I'm hitting the CloudFront, right? So the first time the files is there, not there, so it is taking a time to get that outward. Subsequence time, the file is there, that's why it is faster, right? Either one of the case. So why there is a speed difference for this? How your mid-tier is helping filling files from your... So in the without mid-tier, if I follow my bytes from my curl command, I will go first to CloudFont. CloudFont will tell me, oh, I don't have this file. So CloudFont will go to my mid-tier, to without mid-tier, will go to the S3 bucket and say, yeah, give me this file, the S3 bucket, yeah, okay. Come back to CloudFont and come back to my browser. So my bottleneck here is actually my speed between my Singapore CloudFont Edge and my S3 bucket in Virginia. So the bottleneck is on this stretch. So it's not the speed between my home connection and CloudFont Singapore. I hope it's near my gigabit, else I blame my ISP for that. When you do internal in Singapore connections and then when you go outside accessing something on S3 in US, the speed is not binding completely. I got it, but I'm trying to understand it. So in this one... The other thing is mid-tier is doing, is predicting the next chunk and pre-caching that in CloudFont. So he has a catching component. It's not just the EC2 is employing a magic. No, no, no. Oh, yeah, sorry. I wasn't clear enough. So what's magic here is actually that since I requested this file, my mid-tier server knows oh, I will be requesting this one soon. So while I'm downloading this one during my four seconds here, it's already fetching this one, which will take roughly about four seconds, something like four seconds, 40 minus... Yeah, four seconds roughly. So when 10 seconds later I request this one, it's already in my EC2 instance in Singapore, so I don't have this bottleneck between Singapore and US anymore. I'm just local to Singapore at this time. Exactly, so which means your EC2, there is a catch component which actually are pulling using the prediction time, then it is going to... So the magic here is because it pre-fetches stuff that are the most likely to be requested soon. So that's... So this question I have. You see this problem only with CloudFont. Don't see this problem with other CDN providers. So actually I stole the whole ID. I hope there's nobody from Akamai here. I stole this whole ID from Akamai because we tried Akamai for a few months and they had this magic thing. Somehow they had two origins and they had two origins. They had what they called mid-tier and they had the end layers. And when I understood that, I said, yeah, but why not try to make something like that with CloudFont? So I tried, I had a lot of discussion with the Amazon people here to try to use only the S3 buckets for that. And at the end, the only thing I was able to do was to use an EC2 instance. So I guess if you have access with other CDN providers and you are able to add EC2 instances in between, it will help same. The only problem is you start to have storage with someone, CDN with Amazon, and EC2 instances with Akamai, let's say, and EC2 instances with Amazon. It starts to be the cost because suddenly your internet traffic or your bandwidth traffic cost will just jump to the roof. And that was one of our big problems with Akamai was that we didn't want, each time we needed to push all our files to their storage, net storage, I don't remember the name. We had to pull everything out from S3. We have everything in S3 anyway. So we needed to pull out from S3 and put in their storage. And actually the cost of transferring these files was really like high compared to transferring the same files to CloudFront or inside the, let's say, the Amazon ecosystem. Was that an Amazon cost or back to my cost? Amazon cost. Everything you pull out from Amazon to something that's not Amazon... So to conclude, whatever outbound traffic that's happening from one Amazon to Akamai is charged. Then when you pull to CloudFront it is not charged. It's charged but less. I don't... The price of Amazon is so complicated you can never predict anything anyway. You just take the credit card from your CFO and say you will see the bill at the end of the month and that's it. And then more than 50 signs we drop the price so he stays happy and CFO stays happy. And then we have some talks and you drop the price even more. So you will be deploying the mid-tier with all the regions using the Route 53 to connect and the Route 53... Yeah, so we didn't use all regions because there's some that are of no interest to us business-wise. But we have like most of them like East and West Coast of US, Australia, Singapore, in a few months, India and Frankfurt and Dublin. So it's almost like we didn't have China at all and like the center of the US need because like East and West Coast covers pretty enough for us. Okay, just to see a little bit what it helped us last year because my business people needed proof that I wasn't working on something useless. So that was only for customers coming from India. So you see that they are going... Mumbai, Chennai, a little bit of Singapore edge. So here we have the cloud-front edges on the left side and all these three edges are going... Let's say that's Lucario. I don't know. They're all going to the Singapore EC2 instance because it happened like that. Perhaps sometimes Mumbai some days could go to the Frankfurt instance. And if you see the hit-and-miss ratio, you see the cloud-front has like a 60-30, 70-30 hit ratio which is really bad. Yeah, I was going to ask how that compared to Akamai. Akamai is better. So they are able to store longer the data but you pay for it. They're a bit more expensive. You can talk them to have the price down but if you're a good talking, they pull the price down. How large total file size on the edge are you looking at? One file is between like 200K and 6 megs. One segment. And how many of those, right? Around 4,000. Like all renditions for one movie, sorry. And then like a few... I don't even remember how much movie we have now. Yeah, I'm curious to know like how much you have... 3 or 4 terabytes on S3, something like that. And I think our cloud-front monthly is around 60, 70 terabytes. Something like that. I'm not even sure it's the right numbers. I'm not even sure I'm allowed to tell them. So by doing this, right, you mentioned one problem earlier when you're using directly a crowd-front. You're having an issue of some of the files actually automatically is removed from other files. I didn't solve that. I didn't solve that with cloud-front. But cloud-front is able to pull quicker from wherever it has to pull from so that it's less an issue. But it's still there. And then you see here what still needs to go to the S3 bucket. Like it's peanuts. It's almost no traffic anymore. So that was just like data on three days on the whole content. So it's not just like I pinpointed the good content. It was just data on a short period of time. But I guess if you make the time longer, it would be about the same numbers. So how can we still improve that? So the S3 replication that came out a few months back would already help. So it means that we could have our data in Virginia and Singapore at the same time with a replication that's automatic so that the EC2 instance doesn't have to pull from the other side of the world every time you can pull an S3 bucket that's nearer. So you again win a little bit of time. Obviously EFS will help a lot so we can have more EC2 instance sharing the same cache. It's for failover. Now if I have only one instance that does that, if the instance fails for some reason, I'm kind of screwed and have this weird call in the middle of the night. And one of the things I was thinking while doing the slide actually was that why do replication, why pull directly from a street to local host? Why not call directly with my curl command on my server the CloudFront edge from where the request came from? So instead of just adding the file to my local EC2 instance, at the same time in one call, I will add it to my local instance and to the CloudFront edge because the customer will probably go to the same CloudFront edge for the next segment. Yeah, that's the end.