 Okay, I think I don't need a microphone here am I loud enough you understand me in the okay perfect Yeah, thanks for the introduction. My name is Harold. Yeah, I'm from Austria not Australia. Yeah That was a little misunderstanding And I'll talk about scaling PEP PHP web applications to get ready for the peak season I Need to turn that on and it's better I work at dinatrace as a technology strategist My my background is actually in databases and web development for many many years Besides that I like to do a lot of other things also Going caving down to mother earth exploring Uh, well scaling web applications, I'd like to focus more on Background and pre requirements for scaling rather than giving advice is how to scale directly because what I saw in the last years when I worked with projects mainly in terms of performance increasing is People had performance problems and when I asked them well, do you have any plans already? The answer that I got most times is yeah, well, we need to scale our infrastructure. We need more servers And they expected that everything's run running faster than well when we looked Deep into the system and analyzed what's actually happening there We saw that this was useless because the the server was was bored anyway had nothing to do so a lot of times it's like that The online shop was too slow and the expectation expectation was to scale the infrastructure to increase performance a Couple of words on performance versus scale ability to show the difference a little bit here. Well This is what we think about performance your power of strength Being fast being being strong a single Server a single Individual can be already very performance on the other hand this is scalability so when we have A Fast systems fast applications and we scale them we get more instances of that We get very powerful on the other hand. This is also scalability, right? but Yes, maybe we get less performance out of that, but still we have a scaled environment, so we have scaling here, but maybe not performance does performance matter at all and we heard a Couple of very interesting things with HTTP to now where we learned about Increasing performance with that and yes, definitely performance is relevant And performance has become more and more relevant over the years Back in those days when we when this was the Amazon website. I don't know if any of you guys remember that performance was more about Yes The internet is slow. Maybe I need a faster dial-up connection because this was more or less The real bottleneck back then well times have changed since then The environment the the entire Ecosystem looks different and especially in the last eight to ten years This has Changed dramatically a new digital customer has emerged out there mainly mobility Mobile devices and just a way of how we use the internet It used to be online shopping e-commerce For many years now the world is changing again. It's mainly used to find pokemons probably But in the Millennium generation in the new generation the the mobile device adoption is already 95% and especially here it's very relevant to have fast connections to have fast applications because 60% say that the most frustrating thing is a slow system or a not working system at all so when we get Responses back. Sorry. We are not available or whatever So on the one hand, it's about making the user happy because a happy user is a returning user a returning user is a Converting user is a user that's that's buying products in our online system. And this is actually what what we want to achieve in the end On the other hand, it's not just about making Users happy. It's also about making them happy Because what does Performance mean for Google Search engine optimization is a very important factor. There is already performance If you have a slow website, it will be indicated as slow in your performance in your Google search results and Up to now they are considering performance just for for Desktop environments, but Google is planning to extend that to mobile applications as well now So performance is becoming more and more relevant here. Yeah, and as I said, not only performance, but Another frustrating things for most of people is when we have systems out there that are not Available at all, but this is also a thing that we can cover with Scalability a couple of words later on that later So scalability is more than just performance. It's about creating a well-sized environment Ensuring high availability to not get sorry, we're not Available screens like before we already talked about high performance and to handle large number of transactions high concurrency a lot of Users Using our system, but not only in the operational Perspective but also to respond to feature requests if we have designed our system. Well, we can Respond much faster to new features and find and fix problems much faster But in a large environment where we have Probably not only PHP running, but other Technologies Java.net whatever there is out there as well databases. We have web servers load balancers Even browsers are becoming more and more relevant because more and more code is Running in the browser. We have the these single page apps that Just fetch code from them from the web server. So where should we start here? or What's the main the most important? Tier where we need to do our scaling well We are mainly here to talk about PHP so The bad news here is if we have different technologies PHP is definitely slower than Java PHP is becoming much faster With PHP 7 now, but still for instance Java would be a faster Technology on the other hand the good news is PHP is not the bottleneck So in most cases, it's not about Fixing performance problems in PHP directly maybe in the application But this is a matter of designing the application then it's not actually PHP. That's slow in an application Well the thing that we saw out there. That's the slowest Component most time is The page itself so And this is not a fake site actually. This is a site that was really once out there Online and actually It's still available on an archive site And when you when you load it you find out that the entire site takes more than two minutes to fully render Well, yeah overloaded pages and So if we have that bottleneck in our application It's no there's no requirement to do a scaling on the server side So we have probably other things to fix first. We have to prepare for our scaling so preparing For the scaling process means to break the application really into small components to have these components where we can find the bottleneck then and Start there with proper sizing with scaling. So on the one hand the front end With the browser the code and the browser not only the code But also the design as we saw before on the other hand then on the server side load balancers web servers caching servers application servers and the PHP engine on our case And maybe here not only the application server, but the application itself Breaking it down into smaller pieces in functionality microservices And for sure the database very important we will hear about that later then and create an environment with avoiding bottlenecks and Also reducing load for certain tiers only send load to certain components. That's really requirement That's really required there I'll give you an example we have a Web application we send an initial request to our web server Apache has mod PHP loaded and We sent back the html to the browser It's rendered there and in the next step. We send additional requests Still talking about HTTP one here. So a little different than in HTTP 2 but we sent new requests to the web server for our images for our JavaScript for our Style sheets and for every single request that we send to the web server We need to start a new process for Apache because it's a new request actually and even for requests for static content We have to load all the modules That are defined for Apache here So we have to load PHP engine for our images for our JavaScript for our style sheets And that's actually creating load on the server. That's not really necessary there. It overloads the server That could be done differently Well, you might say well, yes that the content we can use we have that in the cache anyway if it's designed properly yes, that's true, but note that cached content Like here in a in a magento environment. We see for the request we get the three or four not modified That still creates round trips and requests to the web server Because on the one hand we have the caching where the file is stored in the browser and uses an exploration date and after that date then the new request is sent to the server again, but A lot of caching logic is based on this conditional caching So what does it mean we send the browser sends a request to the web server? Using a hash The entity tag for the content of the file the web server checks whether the file has been changed or not and if the file is still the same the web server sends back a 304 only the HTTP header not an HTTP body and The content itself is taken from the cache in the browser, but we still create requests to the web server and For here that means for every single request even though it's cached we create a new process for Apache loading the PHP module so That could be done more efficiently for instance by using a technology like engine x here and only forwarding requests for PHP engine for PHP requests to the PHP engine fast process manager in that case and all the other requests are served in a very efficient way by the web server here by engine x and In the background here, we could even have multiple instances of PHP fast process manager running depending on the requirement so if we have a lot of PHP requests or Requests that require a lot of processing and the web server is easily able to handle that But we need more instances for PHP. We can do that easily in engine x by defining a load balancer here We have an upstream Allocation block for PHP by the way who is using engine x already not that many okay So for those who are using it you might already know that feature here for the others we define an upstream block where we send the request for PHP by fast CGI and in the upstream block we can define multiple servers and that creates A load balancer for us out of the box. That's a very cool feature here so What we have done here is we have a web server and we have already a Scaled environment. We have multiple PHP worker processes running here and we don't need multiple web servers just Because of like a patch is doing it. We have the module to be loaded in the web server Okay, so we have application. We have an application now We we have the web server. We have the application server We have multiple instances running here, but is there maybe a requirement to optimize our application first Because There is this philosophy that I've heard a lot of times for developers Don't focus on performance focus on functionality Performance can be optimized can be tuned later on but I don't agree with that. I always I'm a fan of doing my homework first and doing my work properly and then Trying to see what we can maybe improve more a good example or a good comparison that I always like to Do here is The with with mice so imagine we have a house We have a lot of mice and the requirement is we have to catch more mice because Our mouse catching performance is not good enough well Given situation is so far. We got two cats cat one cat two and In that example if we plan to scale our environment Horizontally here and we get ten of these cats. I think we are still not able to catch a single mouse here so First task before scaling is optimize the cat Make sure the cat is really able to catch your mouse efficiently and scale that cat but Focus always on what's needed don't don't over up don't over optimize. So really Focus on what's required and the 80 20 rule. I think is really a good rule here. So 20% of efforts can solve 80% of the problems and I think that's a good balance here a couple of examples that we had with Where we saw that where people tried to solve problems just by scaling without optimizing their code first is We had Performance bottleneck in a php application in a php environment where we saw that most of the time really was Spent here in the php tier By analyzing that in detail we identified three major bottlenecks on the one hand there was a less CCSS preprocessor on the other hand a social login module and slow php Execution because of a lot of compilation time. It's not that long. It's two years ago, but they were still using php 5.3 and so this was very easy sorted out they just updated to 5.6 and The the compilation problem was not there anymore and even the other two problems Were found pretty easy. So on the one hand most of the time of the response time was definitely spent in these less see Library They Did not recognize that before because they just installed it and and hoped that it would solve their problems with Style sheets. So let's see is a is a style sheet preprocessor, but the problem was they were using it in production environment, but in the development mode. So development mode means they always Reprocessed the the source files, which for sure caused a lot of response time and The other problem was with the the social login module they Had external calls to Social platforms that were not even activated in their environment. This was a bug with that module after that was fixed a actually most of the performance problems were already sorted out and Without doing that we they would have created a Horizontally scaled environment with that bottlenecks in but nothing would have become faster and Yeah, this is the third example with the compilation time for a PHP So in the end what they did after that By the way an online shop Magento online shop they had this small environment and Horizontally scaled In AWS by using a load balancer here The cool thing is in such an environment. You don't need to create a static Deployment here, but you can really react on on requirements of deployments are no longer static Traffic can be low in the morning can be high in the evening can be In a different region during the night so you could Start or shut down instances move Instances to new other availability zones that can be done very flexible in in AWS On the one hand by integrating Metrics directly from AWS, but you have also very cool features to create your own scripts and Use metrics from other monitoring tools where you measure response times for instance or request rates and Up-size or down-size your your environment in AWS That's what they did so they had an Sizing of ten Instances for PHP so a web server here then PHP instances some databases in the background and After It was last year in the peak season for before Christmas and they started a marketing campaign and upsized their environment and started We we had a screenshot here where they ran 48 instances and In the in the beginning everything seemed to be pretty okay law of instances traffic increased and then crash again But what we saw is it was not PHP anymore, but the problem was the database So what happened on the one hand we saw that for one single web request We had around four thousand database statements and Actually, that's not The worst number. I've seen here. We had examples with 20 30,000 database requests for one single web request So where did they come from? on the one hand we talked about that earlier the Entity attribute value System of magento that creates a lot of requests in the background where we when we access one single product on the other hand We saw that the the slow statements actually came from from a Different library with that was used within magento and this was the minify CSS library We had insert statements here that took 30 seconds and more Which makes obvious why? Single web requests crashed at all or and the system was not usable anymore The problem actually was that in their environment when they had Here the load balancer and all the PHP instances running They forgot about a database in the background the database was still designed for the ten instances now they had 50 instances a lot of Requests sent to the database overloaded database Connections became slow statements became slow then with the Update and insert logic in the minify CSS library. They had table locks and in the end the entire system Was not accessible anymore So for database access, it's very important to size the database To Not for the for the for the standard environment really but but for the peak database has To serve all the requests on the other hand To reduce connections and use reduce SQL statements In the example with the minify CSS so what they wanted to The goal that they had with the minify CSS was for sure to speed up the front end Well front end Got a little bit faster, but the back end became much slower. So Actually Cost and and revenue was not Well balanced here just by reducing that minify CSS library the number of SQL statements Got down to almost half the value of before and the system worked again and Especially for databases use caching a couple of words on caching caching is not equal caching, right? We already talked about client side caching a little bit with the That progressed that are still sent to the web server Now a little bit on server side caching Well server side caching itself would be a topic for 10 talks so I Focused just on One aspect here and this is using memcached in a in a PHP environment the requirement that I had is Is coming from from a mapping application so we have an application geographical information system that's using Open maps and this place certain data on the map Certain locations certain distributions of geographical data and in the original version actually all of that data was processed directly from the source data from survey data land survey and Displayed here on the map Well, that's very time intensive to Create that data out of the box always on the other hand It's not really necessary because on the one hand we have the source data on the other hand They are not changing regularly. So when we pre-process that and store it and Make the pre-processed data available for the for the front end that would speed up everything Significantly, and we did that with memcached. So we had a process A batch process running in the background that created a geojson file for the the relevant data and Stored that already with a given access key The key for the URL that's used By the web request and and use that to store it in a memcached server. So This was not done in a web request. This was really done in a background job and The logic to retrieve the data Can be done in engine X directly. We don't even need it don't even need PHP code for that so we define a location block for our get Geojson and Forward the request similar to what we did before where we forwarded the request to the PHP engine Before what the request to the memcached server As a fallback here if we don't find An entry for that key we can always forward the request to PHP to get a on the fly Processing for that data, but in general that should not be necessary So this can be done by engine X directly. We don't even need PHP code for that and Actually when we did that Even maybe before most of us knew about the term we already created our first microservice architecture so we really split off existing functionality from the big application and Created a Certain service for certain functionality and in that case if we have split this functionality in smaller services It's always easy to scale where it's important if we have a slow service We can work there. We don't need to touch the other thing if we have such a large environment with Browsers caching layers web servers applications databases There are a lot of tools out there where we can do proper monitoring. It's Especially in terms of performance very important that we also monitor that because we need to find out where it's slow To be able to react arm on the slow parts The problem with a lot of tools out there is if they are just in the browser just caching monitoring Just web server just PHP is that we cannot bring everything in a in a complete picture And this is where application performance management Finds its role with application performance management we really start at the user to see what the user is doing and already show Is the user satisfied or? Frustrated or is it is a user tolerating the request by considering response times by considering errors that we get and from the user We start following. What is the user doing? What is the user doing in the browser? What is a response time for a certain user action in the browser? And what does this finally trigger on the web server or in the application? So for instance a certain click in on a certain button in the browser causes this database statement causes that exception on the web server Not only seeing what the code is doing, but also We are finding out how Processes are performing how the web service doing how a PHP engine is doing how load balances are are performing what what is the database doing and Not only technical aspects, but also business relevant aspects. So for instance Some examples here from from magenta dashboards. It's then very easy to bring Revenues to bring conversion rates in direct Conjunction with performance with response times with errors. So if I have certain errors in a Certain browser or in a mobile device does this directly affect my revenue my conversion rate Yeah, well, that's it from my side. Thank you very much Questions That's actually a free version for for developers so no cost at all It's yeah, you can find a link here if you want to try it Here on my last page of the slides just downloaded install it Yes, I mean New Relic is one of our main competitors in the APM world just a couple of words maybe on the main difference is That especially for developers we have the benefit that we are not doing snapshots But we are capturing all transactions. So there's nothing you would miss you would see every transaction and Really from from the click in the browser down to the database. We in the newest version We even have an extension here to go into the database and show the execution plan for For the sequel statement and also go down on code level Yeah Certain tears in the application It's an agent-based technology. So for instance for PHP it's an extension that you load into the PHP engine and then we can Trace what the execution is doing there. That's different from technology to technology. So for for a patch. It's a module for for engine X. It's also a dynamic module that we load Even in engine X version that do not support dynamic modules, but we found a solution for that Yeah Yes, I believe yes, thank you Michael. Thank you Last speaker for tonight