 And I'd like to introduce Max, who's going to tell us about using Uwizgi, which I've recently had a very good experience with, so carry on my hand. Thank you for coming to this talk. I know it's hard to get up this early. You know, this beer party is still the midnight every day. So, yeah, I also had problems today. Okay, first the question, who is already using Uwizgi? Okay, that's nice. I know that Roberto is here. Roberto, yeah, this is the lead developer of Uwizgi. Can we give him some applause? So, the talk is called How We Switched Our 800 Plus Projects from Apache to Uwizgi. So, who we? I work for the company called ILOVE. It's a web agency, Russian web agency. We do websites mainly. We are the part of ILOVE Group, which consists of three companies ILOVE, ITARGET and ICOM. ILOVE is pronounced just like ILOVE, not AILOVE, because some people ask me about it. I don't know why, don't ask me. So, we are situated in Russia, in Moscow. We have more than 120 people, and we made more than 800 projects during the last years. I personally work as a team lead for the ILOVE company, and this is one of the largest Russian web agencies. Okay, Hawaii. My name is Max. As I already said, I work and live in Russia, Moscow. I'm also the author of Python RedMine package, which is currently the most popular and feature-complete package to access data from RedMine from Python. Architect, probably, you've seen my lightning talk on Monday, which is a successor of Django DB Party package, which provides fully automatic table partitioning for Django, PB, Pony, and SQL Alchemy users. When we just started, we used only PHP for developing our projects within GionX on the front-end and Apache with mode PHP on the back-end, which was the best available way to host PHP applications at that time. It was a long time ago. But then, after several years, we realized that there are a lot of problems in downsides with PHP itself. So we started to search for another language and choose Python. We chose Python for several reasons, which are beyond the scope of this talk. And because we already had already working in GionX and Apache environment, so we decided to choose between mode Python, which was completely dead and abandoned at that time, and mode Whiskey, which was also dead, but its corpse was still warm. And, yeah, you know, it was more feature-rich and had more documentation. So we chose mode Whiskey and started to serve our Python applications with mode Whiskey. We used mode Whiskey in demon mode because it allowed us to run applications in separate processes and was also the recommended way from mode Whiskey author himself. Actually, everything worked, but we had several problems with Apache and mode Whiskey. Okay, our main problem was that we had projects written in different versions of PHP and Python, and the only way that we found we had to have five Apache instances one for each version listening on different ports, which was really nightmare to support. Okay, another problem was with automatic code reloading during the development. We used Django primarily for our projects in Python, so we didn't want to use Django's built-in web server, and we wanted to develop with mode Whiskey because we wanted to be sure that everything will work as expected after we deploy on our production service. And there are two modes in mode Whiskey. Basically, one is called embedded mode, and as far as I know, one needs to restart the whole Apache after the code changes. And there is a demon mode, which we were using, and you have to use your touch command to the Whiskey entry point to reload the source. Also, there was a recite from mode Whiskey author, which was hidden deeply in the docs, and it was the 100-plus-something lines of custom-made Python script, which he wrote, and we had to include the script in our projects and live with it. Yeah, I know maybe it's not a big problem, you know, but still, it's, you know, 2014 year, so there must be a more proper way to do that. Actually, at the moment, an active development of mode Whiskey started again, and there are several new versions already, and started from 4.1. There is a Django application plugin, which adds a run mode Whiskey command to the Django, which allows Python, you know, manage Py, run mode Whiskey, reload and changes command, but that's only for Django, and it's actually the same old 100-plus-line script, which is now just became the part of official package of mode Whiskey, so nothing new. Okay, another problem was that during development, we created a lot of different branches in Git, and we want them to be available, for example, you know, via URL like branch.project.com, and we also wanted each branch to run on the separate demon process, because if one branch has some problems, and these problems can crash the demon process, the other developers should continue to work, so the solution we came up with is to have a Git hook, which generates a separate Apache config file for each new branch, and then we start Apache. If a branch gets deleted, then the config file is deleted too, and we have to reload Apache again. Now, okay, if we have 20 branches, we have 20 almost absolutely similar config files, except for the different demon process name and a few other things, and this is only for one project. So what if we decided to change something in the config file of a project, we have to edit all these 20 config files, you know, this is really painful. Perhaps there is another way, and I hope there is another way, but we spent almost a month googling and we couldn't just find it, so maybe if somebody knows, please tell me. I'm really interested. Yeah, another problem, more problems. So when we add a new config file, we have to issue a reload because Apache can't load configuration files dynamically, but there are actually two problems with that. First, there is still a little time when this web server don't respond to requests. This time can be from milliseconds to seconds, depending on various factors. And second, if there is an error in your configuration file, the whole server will crash. Yes, there is a config test comment which can check the syntax of the file, but this is only the syntax, so if it's not the syntax error but some logical error, the server will crash. So this is not good. And lastly, some small problems, maybe subjective, but still. First one, Apache configuration files are ugly. I know a lot of people, and me including, who thinks that. If you look at the Apache configuration files, you need some time to understand what's going on in there, especially if there are some complex ones. So Apache is hard to configure properly. Oh, sorry. You have to be really Apache expert to configure it in a way that it could compete with other web servers in areas like memory management, CPU utilization. So not every sysadmin knows how to do that. Yeah, Apache is old. You know, even my grandma was using it in the old days. Yeah, the 2.2 version was released in the 2005. Yes, there is a 2. Yeah, it's still developed this branch, but actually they mostly fix bugs. And 2.4 version was released in 2012. And yes, it's fixed a lot of problems and limitations. For example, memory management became much better. They finally added an ability to declare config file variables. So config files became a bit cleaner too. And yeah, modWizgi seemed to be abandoned at the time when we were looking for a solution, but it's actually developed again. So this is not really a problem now. And all these problems made us to start looking for a solution and quickly found the solution. It's called QWizgi. So QWizgi. It's a modern project. If I'm not mistaken, it started somewhere between 2009 or just 2010. It has a really fast development cycle and new features are constantly adding. It supports a lot of languages including Python and Cpython, PyPy and Jython. It supports speech, Pulu, Perl, Ruby, Erlang, Go, Java. It has support for V8 engine and Mono for running ASP.NET applications and more. So it's really cool. It works on all Linuxes, Windows, PSDs and other OSes like Mac OS, Solaris and so on. Jynix supports QWizgi protocol directly and QWizgi is the best performing protocol of QWizgi. It also has a ton more features. We'll discuss some of the most interesting later. And just a brief introduction how we can install it. It's really easy. So you just go download this latest version of QWizgi. The vast majority of QWizgi features are available as plugins, which is also really cool. And if you want to have the maximum amount of flexibility and use only the minimal amount of resources, so just create a modular build, which is the recommended approach. I'll show how to do it from the source because OS package repositories may not always have the latest version or may not have the available plugins. So we just download this QWizgi thing. If you want to change install location, you can add the two following directives to the buildconf slash core.ini file. And then you create the directors, of course. Now we have to build the core. It's really easy. Core already consists of several packages that are most likely will be needed by everyone. But if you think that you're not everyone, you can also customize what exactly the core will consist of and recompile it. And then we needed to build our plugins. Two plugins for Python, 2.7 and 3.3. And let's build two plugins for PHP. I know we hit PHP, but still. Okay, we just completed the install process. It's actually very simple. Now let's see how we can solve problems that we had with Apache and more with QWizgi with the help of QWizgi. First, yeah, multi-version plugins. This is really cool. This was the biggest problem for us. And this just works in QWizgi. You compile as many plugins as you need. And it just works. Yeah. Remember that 100 plus lines Python script, which was used in modWizgi to monitor core changes and real-trilode demon process. QWizgi has just a simple option. You just specify how many seconds you want this QWizgi to scan for your core changes. And it reloads everything automatically. There is also a touch-reload option if you need it. So no more custom-weight home scripts, which you have to search and include in your projects. Finally. We also had a problem with modWizgi's with QWizgi demon process directive. Name can't be dynamic. And we needed to separate demon process for QWizgi branch. So we had to generate absolutely similar config files. QWizgi has a thing called emperor mode, which was the solution to this problem. Basically, it's a special instance of QWizgi that will monitor special events and will spawn stop-reload instances known as vessels on demand. By default, the emperor will scan specific directories for supported configuration files. But it is extensible using imperial monitor plugins. That means that you can store configuration in Postgres, Manga, publish it via MQP0MQ, and so on. There are many options. Let's see an example for the directory monitor plugin. We start a QWizgi emperor instance and ask it to monitor the following directory pattern. Now, if we add a new file to this directory, the emperor will automatically spawn a new vessel. And if we modify the file, the emperor will restart a vessel. And if we delete the file, the vessel will be killed. And all of this will happen absolutely automatically. There is no need to issue some. We start commons, reload, and so on. Now, what if we have an error in our vessel configuration file? Actually, nothing bad will happen to the other vessels. The emperor will just say that this vessel is cursed, you know, and won't spawn a new instance of it. So if the emperor dies himself, then obvious that his vessels are died with him. So how will this actually help us with this Apache problem with dynamic Wizgi demon process problem, yeah? Should we also create the same insane amount of configuration files, the project, one for each branch? Yes and no. We'll use template config files and create Simlinks. This is an example of template config file. At first, we define our own variable called projectDue and set here the path to our project. Then we define that we'll be using Python 2.7. This is our plugin. And our other Python project-related stuff, which should be nothing new to anyone and that's how Python apps work. The interesting thing here is the person-to-an-magic variable, which is the config file name without extension. By the way, I don't know about you, but for me, this syntax is much, much cleaner, you know, compared to Apache configuration files. And also as a bonus, my brain doesn't hurt after I work with these configuration files. So it's nice. That means that from now on, instead of generating separate config file for each git branch, we just create a Simlink to the project's template file. And yes, this will give us 20 Simlinks for 20 git branches still, but the big win over Apache is that if we need to change something, we change it in one place in the template config file instead of changing every config file generated for Apache. And also because we're using the demon mode, which is monitoring our directories for config files, when a new Simlink is created, a new vessel is spawned automatically. And if we delete this Simlink, then the vessel is killed also automatically. No need for any restart commands and so on. Also starting from 1.9.1, you can now tell the emperor to spawn a vessel only after the first request has been made, combined with idle and dianidle options. This allows us to have really truly on-demand applications, which is also cool. Okay, let's briefly talk about some of the Ubizgi's interest in features. So we can implement autoscaling with Ubizgi using the Brutelord, Zerg, and emperor modes with idle and dianidle options. So it's a combination of features, you know. The idea is that when the site load is heavy and your vessels just can't handle it, they can ask emperor to enter Brutelord mode and give them some help, Zergs. So they could win the battle and serve all the requests. And after the load is normal again, the Brutelord kills all the Zergs and everything is fine again. As of 1.3, there is an alarm subsystem. It allows the developer or sysadmin to announce some special conditions via the various channels. For example, you may want to get notified via jabber when a string terrible alarm appeared in log files, for example, or something. There are a lot of also options to configure this alarm subsystem. There is also a nice feature for aliasing Python modules, you know, that having multiple versions of Python package and model is very common. And the one way is manipulating Python path or using virtual ENFs. But, for example, Yuizgi gives us another option called aliasing system. Let's say we have imports of full-in-bar modules in a lot of places in our code and we want to make some modifications to it but keep our original full-in-bar module syntax for whatever reason. We can create experimental full-in-bar modules and make changes in them and alias them and when, you know, you make an import full-in-bar instead of importing the original files, experimental full-in-bar modules will be imported. There are a lot of cool features. We need a lot of time to, you know, talk about all of them, but briefly, there is also a built-in Chrome tab. Yeah. There is load balancer, clustering subsystem, offload subsystem, which allows you to automatically delegate some heavy tasks to separate threads. It also has a lot of different plugins for different tasks and it integrates with almost all well-known web servers like, you know, and so on and so on. It has also Django admin integration plugin which shows the status of Ubis-G and allows to restart or clear its cache and there is more functionality in development. Also, it has a rich configuration system. It supports configuration files in any XML, YAML or JSON and so you can choose what you like, what is better for your brain. It has more than 30 magic variables for all sort of things, you know, like environmental variables, placeholders and so on. You can even do simple math in placeholders. You can read contents of other files from your config files. You can write four cycles and deep statements in config files. You can declare your own variables and use them in Ubis-G and much more. It's cool. Okay, finally. How to switch? Ubis-G is cool, yeah, but how? You know, there is no easy way, unfortunately. There is no some magic tool that will translate all your Apache config files to Ubis-G once. How we did that with our, you know, more than 800 projects? Well, we divided our projects into several groups that have equal or almost equal Apache config files. Then we wrote a script that generates similar analytics for Ubis-G for each group. They took us approximately two days and allowed to switch all our projects to Ubis-G on our dev service. Then we started to run our functional tests and see if all of them passed. Then whenever there was a problem, we just looked at it, and, you know, this way we tuned the config files for Ubis-G. And when we were sure that everything works on our dev service, we made a switch to Ubis-G on our production service. So that's how we did that. Again, everything by hand, unfortunately. Conclusion, yeah. Conclusion, I want to say that Apache is actually not bad. It's a very good and stable web server which is used by a lot of people. If it's suitable for a lot of situations and you can happily use it if it works for you, do not blindly trust people that say Apache sucks. Go try this or that new super cool web server. Just tune on your brain and think, do you really have any problems with your Apache? Do you really have any problems with it and don't listen to anyone? In this talk I explained why we switched. So we didn't have any problems with the Apache performance, but we had problems with its features. So we could do everything we wanted with Apache, but we wasn't happy with how we could do that. And what I'm trying to say is that you should choose a web server by features and not by benchmark, though Ubis-G has a much better memory management than Apache. So you'll probably win something in performance. And also it's pointless to benchmark a web server because usually the old perform approximately the same if they configured properly. And in 99 cases this is an application that is written incorrectly and has performance problems and not a web server, you know, which just serves an application to the world. So thank you. Thank you Max. We've got time for a couple of questions. Anyone? Yeah. Pardon? Come here. Compared to Ubis-G with Apache, but one is I guess an application server. The other one is a web server, so isn't going to be compared to Ubis-G with more problems than Apache? Yeah. Yeah, yeah. So the question is if it's maybe was better to compare Ubis-G with more Ubis-G, you know, and not Ubis-G with the whole Apache thing, yeah? Yes, yes, of course. In front of... So we had Gionics in front of and Apache with more Ubis-G on the back end and now we have again Gionics on the front end and Ubis-G on the back end. Speaking about the comparison, if it's a fair comparison, I think yes it is because mode with G is just, you know, a thing to serve Python applications, but it just won't work without Apache itself, so of course. And well, I don't know, I think the comparison is fair, yeah? Yes, you can use Apache as a reverse proxy. Yes, you can, but you know, the problem was that it's not very comfortable to work with Apache. You have to search for some work-arounds always to make everything work. We didn't like it. One more question? About just to go to the flow about the one could have that the U.S.D. could be compared to Gionicorn, for example, and in this case you could have a setup with Gionics and behind you could have Gionicorn on Uwiski equaling so Apache is basically to be replaced with Dynex and behind this you have an application setup which could be Gionicorn to be used. Yeah, but Gionicorn, it can't serve any other type of applications, yeah, only Python. Yeah, so it's maybe you can compare Apache or their full web service. They support a lot of languages, a lot of technologies and so, yeah, this is why I did this comparison answer. We will stop now. I'm sure you can answer any questions with the corridor track. Okay, thank you, Max. Stop there, Tom. Stand up, please.