 As we're all settling in, hello and welcome to my talk. I talk around about CDNs and beyond. We cover how we can speed up websites beyond just having a US or Europe-centric focus. I truly appreciate that you're still coming to sessions. Has been three days full of sessions and a lot of things. Only one or two guinesses in the evening and other things. Are you all having a good time so far? Yes, great. The alternate title of my session is Make It Fast. Who of you likes to have fast websites? Hands up. Who managed it to do it even if you're sitting in a train? Not that many. But if your people are, let's say, in India or in China or in South Africa, then it gets a little bit, oh, thank you. You can join me on stage if you want to because you're pretty settled in. I'll talk about this. I'm Bastion, working at the Macy I.O. as a systems engineer. For the Drupal world, I'm contributing or contributed to the Drupal CI, which tests all your patches which you upload. That was my contribution to it. For the agenda, we start with a small introduction. Then we touch base with the key facts of fast websites. We're also covering several CDN solutions. We then segue into hosting in China, which is probably really interesting for you. It comes a little bit late, but I felt I need to give you a proper introduction in what can be accomplished and what parts need to be optimized to really run in China. Because if you just start in China and then you didn't look at all the key factors, then you start redeveloping everything. I touch small parts of the cash warming we do for the China caching infrastructure and then some takeaways from the whole presentation. But what makes websites fast? Some basics. You need to keep your website, the footprint, the bytes you need to transfer from your server to your end-users browser. May that be a mobile device or a desktop computer or a tablet as small as possible. You need to keep the requests pretty low because if you have a lot of requests, it takes a long time on bad connections to just go back and forth. The end-users look-ups is also something. They should be fast. It also depends where your server is located. So if you use, let's say, New Zealand and your server is somewhere in the Arctic, it takes quite a long time to get the data over. Bandwidth is an issue and caching on the client side and also on the server side. But you might ask, why should I care? Well, you've probably seen that quote quite a few times. So Amazon found out that every 100 milliseconds of latency in page load time costs them 1% of profit. Just keep that in mind when you're developing a website or a product where people spend money on, that if it's fast, you have them. They will like, okay, I want to buy that. They click, it loads, it loads, they fade away. If they click and it just loads right away and they can pay you, then you've won. So fast websites also have a good turnover. Another really, really good talk is the website Obesity Crisis from the guy who's running Pinboard.in. And he looked at the size of websites over quite a long time. And when he started to write his talk, a website was around 1.8 megabytes. And he just extrapolated the data he found and said, hmm, at some point we will reach 5 megs for that site. And he just waited some years and went back to that website. They did a redesign. The content was still the same, but they had like added 2 or 3 megs of data which has to be transferred in between. He then came to a really interesting tweet. So he said, he proposes that your website should not exceed the size of major works of Russian literature. Because if your website is more than 1.8 megs, if you're targeting countries which aren't that fortunate with bandwidth, you should probably scale your websites down a bit. I'll touch on that when we are getting back into the whole China thing because their bandwidth is something which really is different. I mentioned you should keep your number of requests pretty low. So bear with me. So less requests mean you need to do less connections to the server. It means also you need to do less round trips back and forth, and that equals in the foster website. With Drupal you do that, for example, if you do the CSS and JavaScript aggregation. So all your files are compacted into one file and you just do one round trip. So DNS lookups is another thing. If you look how a website is transferred from your server to your end device, it starts with DNS resolution. And before we even get to the server, we need to find out the IP address of that given server. So that's 56 milliseconds, and they're gone. You can't do something with it. They're just the device asking for the IP address of the server. It can be if you have your DNS server located somewhere in the states that it takes some time to get there, get the IP, and go back to you. So you can do optimization on that part. I did a little bit of different thing, and I traced a HTTPS connection. So that's a really good close DNS server. It takes five milliseconds. That's pretty darn good. Then we open a TCP connection which takes around 30 msx. And then we do the SSL handshake, and that's where things get really ugly because it just takes 176 msx, and you can't get around them. It will be probably a bit better with HTTP2, but as you probably heard in the talks earlier, Human or Fastly was speaking about HTTP2, probably it's not... HTTP is not what you're expecting to encounter. Then we arrive at the server, and he gives us an answer of 33 msx. That's fast. You might wonder why the content transfer is zero because it's a redirect. So it was just a header coming back, so the transfer was pretty small. Settle in with a good DNS provider, so... That's interesting. You can roll it your own for fun and profit, but probably it's not that easy and not that sustainable because, yeah, who had problems in setting up C names on his root domain? Quite some. Yeah, that's the thing. By design, you cannot have a C name pointing to your root domain. There are some providers which just go over that RFC and find solutions. You can settle in with DIN, NS1, or go with Route 53. They all support it in more or less good ways, and they all have really good geolocated DNS. So your request will end up quite near to you. So sometimes we also talk about server location, and the discussion in the end at some point ends up with me saying physics. So given an example, we have a big customer in the States, and he has a user base in Switzerland, and he says, oh, the site is slow, I see it on Google. And then I say, yes, of course it's slow. All your traffic needs to go back to the States. So bear with me when I go full-on physics with you. C, the speed of light in vacuum, is around roughly 300,000 kilometers per second. For the moment, sadly, you can't get faster than that. So to do that, that would be awesome if you can do it, get back to me, because we should start a business on it. That's where CDNs come into play, and the CDN is basically moving the endpoint of your web server closer to your user. That helps a lot, because you can cache the full sites of your user near them, so they don't need to go back to the States. If you're here, you just have probably a point of presence in Dublin where you get the traffic handled on. Caching, so I guess nobody would survive without varnish. Say hi to varnish. I guess everyone is using it, and we discussed yesterday during dinner what would happen if varnish would decide that they go on a full-blown, just pay-per-use product. I guess we would all be pretty unhappy about that. So varnish saves our life, it caches on the front of your web server infrastructure, and that helps you a lot. But how about client-side caching? Our infrastructure, which I run, we try to set long-lived headers as much as possible, because when you do requests over and over again and you go to the start page, and then you move to another page, you don't want to reload all the assets you had in your site. So that's helping quite a lot. There are a lot of really good resources, and I put them in the end of the slides, which you can read through and probably find solutions and ways to deal with the request headers. Actually, yeah, CDNs can help you, and they are pretty everywhere. So that's the point-of-presence map of Fastly. Everywhere you are, or nearly everywhere, there's a big void in China and Russia. Yes, but that problem's solved, you know, because we have a cash note there. So everywhere you are, it tries to find the closest point-of-presence to you, so it speeds things up. And that brings us to CDNs and beyond. I work with several vendors. One is Fastly, Cloudflare, or KeyCDN. You can also go to Akamai, but then you need to have a big company credit card because it gets really expensive at some point. I'll just touch base a little bit on the pros and cons, which all those providers have. Fastly is really good. You have a pay-as-go payment. It's built by requests and the bandwidth you use. Basically, you can think about it as a distributed varnish as a service. So they have a lot of point-of-presence where they just have a lot of varnish power, and you can just ship your VCL to it, and it will make your sites faster. You can do full-page caching, and they also do DDoS mitigation. SSL, on the other hand, because we're moving into a world where we are having a lot of SSL going on, and Google is also ranking you lower if you don't have your site available by SSL. SSL is coming with a premium for Fastly, but as far as I heard this week, they're working on it to revamp their pricing structure a little bit to make that work. The other one is Cloudflare, and I usually refer to it as nobody ever got fired for using Cloudflare because it's a freemium thing. You can start with it. It supports SSL. They can do DDoS protection. It's kind of proprietary because in the end you're handing over your DNS server to them, and I'm not sure, but in information technology you don't like to put all your X into one basket, so I'm not liking that too much, but for some site it's really good to use it. Another one is Key CDN. They also have a pay-as-you-go price scheme. I started to use them because we have a site which has heavy traffic in Switzerland, and we wanted to make leverage of a CDN. The problem is most of the CDNs aren't in Switzerland, so the nearest pop would be Frankfurt, and our servers are in Zurich, so you move traffic back and forth, which is not good for performance either. We started with them. Sadly, currently they can't do full-page caching. As far as I heard, it will be possible in a few months, so we have another competitor in that market which helps you quite a lot to do that. Currently it's perfect for delivering assets because they come with a free SSL certificate for all your assets, so you can leverage that. But how about other countries, you may ask? Well, let's talk about China, and that's where the things get really interesting now. So, basically, unlearn everything you know about the Internet. That is basically a joke now. So, China is different. So, to do this, get a partner in China that may be your beloved client who has an office there because you really, really need them. As an European, you can't just go there and say, hey, Amazon, I'd like to have one server. Can I get one? They are like Chinese, and they basically say, well, we give you servers, but you need to have an office somewhere in China. You need to speak Chinese, and it's getting a little bit complicated there. The other to do is if you have a customer base in China, you need to have a server within the mainland China. You need to be prepared to rethink how you built your site, and you need to obtain an ICP license. Oh, let's go up now. An ICP what? So, yeah, Wikipedia says it's a permit issued by the Chinese Ministry of Industry and Information Technology, basically, and what it is, it's that. It's a license plate for your website. It needs to be on every website you put out and host in China, even if it's just caching. So, if you go to Baidu, it's that string on the bottom, and if you have that in, then everything is good, but if you don't get an ICP license, they will shut you down. And that's something I was like, wait, I can't shut me down. I still have my server, and it actually happened to us. So, they shut down our site. Monitoring went red. I was looking on it. Server looks fine. I can get on it. Checking the main IP. I was like, what in Chinese was coming up? And I was like, what? So, basically, what they do, they have, every hoster has a big nut router, which they put my IP in front and some web server in the back, and I get just access to the web server. And it looks perfectly fine as long as you connect by SSH over the same IP. But if you do web traffic over it, it says, oh, well, sorry, Chinese government said no. So, that's a thing. We started looking at our website and did some tests, and we started to find out what we did wrong to host our site in China. So, we had web fonts loaded. Connections from inside Mainline China to, let's say, Switzerland or the US are capped at 15 kilobits speed. That's pretty slow. Have you ever thought about how big your web font files are? Even if they are 300K, it takes quite a long time to transfer it. And then there is that huge, wonderful key image you have on your front, which is also optimized that somebody can go on with his iPad and it's like two mechs. And your site is loading and loading and it's just that key frame visual you have on the top. Another thing is, host your JavaScript libraries on your own because you also don't want to have connections going out to it for. Getting rid of external JavaScript dependencies is also an important thing. And don't try to do HTTPS, please don't. Or get a license, but that costs also a lot of money. So, try to get rid of them that too because they will shut you down if you have HTTPS running. Fun fact, we have a monitoring site which is on HTTPS and that got shut down too because, yeah, but why you may ask? Well, that request or that request or that request, it's all in your websites. And while it goes to Google APIs, the Chinese and Google, they aren't at friends with each other so they cap or stop your requests. Twitter? They have their own Twitter. Why would you even think about adding a Twitter widget? So, the thing is and the thing that makes you your day and your ops people like the day of your ops people really bad is the great firewall of China. It's the thing you don't really, it's not something that works the same all the time. I always refer to it that firewall is blocking your requests maybe sometimes a little bit, sometimes it throttles it and it's really unpredictable. I have a slide quite a little bit later which also shows you how that thing works. So, friendly advice if you host in China get a separate domain because you don't want to have all your content on one domain because you need to somehow split your content. You want to have the full blown website with all the Twitter widgets with all the JavaScript ready on one end and you have a stripped down version on your customer.cn which helps you a lot to maintain those two sites and have it fast. Try to strip out everything you don't need. Do you really need a draggable Google maps map or is an image probably enough? So think about those things, try to think about outside the box when you tackle those issues. While you are on it get a Chinese DNS provider too because you have the same issues with DNS requests from within the firewall zone to the outside zone because it's just slow sometimes they don't resolve sometimes it just takes 30 seconds and then you get at that tipping point where all your applications start to fail because all the timeouts are around 30 seconds and sometimes it goes, sometimes it doesn't work. Otherwise number three who speaks Chinese? Nobody. Learn Chinese or hire somebody who speaks it fluently. Because you really don't want to deal with the authorities there and probably they also may not speak to you because you didn't grow up there so you don't have a passport there. Some other people are saying, okay we had a Chinese CDN, right? Well others say, okay, no problem we have a CDN a point of presence in Hong Kong but Hong Kong is not mainland China so it's outside of the firewall zone and we actually did some tests so we did a huge list and searched for providers which are like cloud providers and we got servers and we tried out and even there is like, you could nearly draw a line on the map where the firewall zone is beginning and where it isn't. So as long as you're not within this zone you can't do something. Another thing is the CDNs are pretty expensive. I just blacked out something on that. I asked like, hey, I'd like to have your CDN solution. They were like yeah, sure, it just cost you 12.8k per month and my customer was like wait what? Because it outnumbers your costs of the hosting quite easily and the thing is if you think about your networks or your internet infrastructure in your country you you're probably aware that you need to connect in every country. So if you're talking Amsterdam there is the internet exchange where the providers peer up in order to make your connections as fast as possible and your traffic should not go if you want to talk to a server in Switzerland over the states you can go via England back to Switzerland from here. They don't do that in China. In China every internet provider is actually owned by the country so they think well, it doesn't fast internet why? It's fast within if we peer with each other that would be even faster but that would also mean costs so we don't do that. The fun thing here is you can have two tiers one is the pricey one which basically they are interconnected with all providers and the cheaper one is the top three providers so that's something which you can think about. I don't recommend it though because it's really pricey. The price tag comes also with some services which is good because they may help you getting the ICP license but you still need to have a valid a valid company in the open company which does sell goods in China. The agency in going there and say hey, I would like to do that they say no because you can't speak on behalf of your customer so there's a lot of red tape involved and you don't want to do that. The thing is if you pay money and pay somebody money you also have someone you can blame which is really good when it breaks. To have a server in Mainline China and that's what we currently do is I always say I treat it like having a server on the moon because you can't see the moon all day because at some point you don't see him and in the evening he reappears. So we can't connect to that server all the time and we needed to find solutions how we deal with it. Puppet runs, yeah, they take around 30 minutes and plus and they don't do something they just try to connect to the master we have in Zurich and it just takes ages to get through it. SSH connections yeah if you have an outage you need to probably make a tea first open the SSH connection really calm your nerves because you type and sometimes it starts to reappear there. The connections are slow there is a lot of packet loss and I always say let's get creative we know what we had or what we saw until now but we don't know what is starting to appear tomorrow probably tomorrow they shut down something probably tomorrow half a day we don't have a connection there's fun involved that was the thing I got when they took down the site and I was like wait what? and I got back to a provider or the guy who was owning the server and it was like well I don't know and I was like can you call the government and he was like well calling the government that's really no can we just leave it as it is your site is down so you deal with a lot of things that you don't have when you host somewhere in the country which doesn't have that intranet thing another thing actually that was yesterday morning 6.30 something like that that's IPv4 connection and 4 is good and that's like I don't connect and I don't connect and well the thing what we do is we borrow more caches there and we have long-lived caching so we have actually a cache warmer which starts to cache the site and if we lose connection we just serve stale caches for a long time that's not the perfect way to do it it's the cheapest way to do it we told the customer hey you might not have all the features you want to have on that website but in the end do you really need all those features and we just narrowed it down to features where they say ok we can live without that but we need a web form so for web forms we have other solutions to redirect them to an external solution which works in China and is probably hosted in China because it's faster I already started talking about warming the caches so with 1.3 we had 1.3 play who is using it nobody 1.4 that's ok who is still stuck with 3 and doesn't know how to update the VCLs nobody good well with 1.3 we had 1.3 play and then we updated we started to update to 1.4 and our testing environment broke and I was just lugging into the box as one does running 1.3 play it wasn't installed I was like would 1.3 play not be here it's a part of 1.3 since a long time a lot of yuckshaving later I found that in the changelog of 1.4 they removed it there was no public announcement about that and we were like that's bad because all our cache reformers were breaking so we started looking for solutions and actually what we found is go replay probably some of you are familiar with the tool go replay basically spawns a process and you can say listen to the traffic on your port 80 and send it to another server so what you can do there is you have a staging environment or it's originally built to do low testing on a staging environment you say take the traffic of port 80 and send it to another server so it's not getting in your way of production traffic it just listens and gets the connect CC and sends it to another one the cool thing about this that tool is you can write the request to a file and that's exactly what we do we are running a full side load of every site every day so if content changes it lands into that file and we know the links and URLs and assets which are present and we re-run that on a regular base so we've warmed the cache with all the content we have it's not the best way to do it but it works pretty well because you do it every day and you're good and in the end your users are always having the most recent content on it so how it works for us is we have in China we have a varnish server which tries to connect back to the ancient varnish servers in Europe and if we don't get a connection we just retry it and you need to be really graceful with all your connections because they made time out the gore proxy on the other end is logging all the requests which are fed into varnish and sending it to a file and we just start to warm the cache all the time and the good thing is if we have a hit already means if we have the asset already cached we just go to the next one because we don't need to update it so takeaways you need to optimize your site and if you know you have a project which the customer is really fond of hosting in let's say China or India Russia or South Africa then you need to think about how many requests you're doing and minimize your website to what you really need China is different be prepared also for that it's not just different in a technical way of saying it's different in a way of how people deal with each other your provider may not want to talk to the authorities about the day shutting down your site because it's not how they deal with it so you need to find ways you need to learn how they work what we learned on that way is there is no cheap way of dealing with that so it's really hard you can find a solution to do it but the cheap way is not always the best and get good partners early on also means prepare your client that he will pay probably more for his website in China than he pays for the whole Europe and the US combined so that's one thing we really pushed our client to say okay you want to go to China be aware that you will need to pay money to make that happen of course you can move your site or clone your site completely to China but if you have backends you want to talk to or if you have a login and shop solution then it gets really really complicated yes that concludes basically my talk about how you can deal or how you can get into the Chinese market for a start if you want to grow bigger then you need probably to invest more money to like get proper CDN solutions or maybe have your website cloned there and find a way how you deal with the backend situation because not every project needs it our customer is happy to have like a read-only news site which informs the user base about the latest news the latest products and so on yes questions I'm happy to answer all of your questions if you have so have you worked with the thing called the golden machine in China? the question was I'm just rephrasing the question was if I have worked with the golden machine? no if you probably walk up to the microphone so people can hear it I just heard of it before because of another possible project we did but the golden machine is supposed to be for a commerce site you actually have to send all the transactions information to a governmental server basically okay? no I didn't hear about that we had a lot of things that were like I bought virtual servers and wired the money over to them and the transaction was done and I didn't get a virtual server so I'm usually really really careful if it comes to that because you just don't know and if the other end doesn't reply you're basically you're done the thing is they deal completely different with handing over data to the government if you would ask your customer are you fine if the government is informed about every order we place into your system the Chinese would probably say sure let's do that and your customers know why so that's things you which are a little bit different it's a cultural thing what are the questions? Lucien yeah I'm here today and tomorrow during the sprint so if you want to talk discuss something about it just hit me up I'll be around available for you thank you