 Sveče, da sem vse. Zdaj sem odličil, da sem odličil, da sem odličil. Sveč sem odličil, da sem odličil, da sem odličil. Mi je Davar Gutieres. Mi je v Sloveniji. Se, da je to vse očetno. Mi je spenija, začel naši, koresija, naši, in sem v Sloveniji. Mi je zelo, da se je zelo. Zelo. V Slovenijanom 3GN, v Slovenijanom 3GN, to je način v 1996, je spremljala v Kaptelj, jubjana, nam je 30 tri mplejci. Vse spremljali vznešče ibnimi, zelo, v Slovenijanjim stavli, s ediptevno, in vzloženju, vzlčenu, na vznoščenu. Zdaj smo prišličnih vrtičnjih, prišličnih vrtičnjih v Red Hat Enterprise in Oracle VAM, tako češnjih kvm in Xen solušnjih. Zdaj, da smo prišličnih vrtičnjih, smo prišličnih vrtičnjih, da smo prišličnih vrtičnjih. Zdajso��u smo prišličnih vrtičnjih o te lahko preformavanačno행 en har1000 preform확 공angliske o zra раскub?.. K Grasning Netfacking Skengit kvčnožّskih in tom že malo, predražijo za vlastvenješku. Kaj je optimizator? Kako je naredne naredne servovalo, potrešče je zelo pomembnjel, ale pa neč včasno, kako doben, servovalo je zelo pomembnjel. Zelo pomembnjel je sloven na naredne servovalo. Ne zelo naredno vse. We are using new slasks, new redhead, but it's slow. We must determine what is slow in our server. Maybe too many services running, maybe it's something wrong with disk drives, with IO, with network configuration, and then we must do TCP tuning, network tuning, disk tuning, memory tuning, disable unneeded services and such a stuff. What is performance? To boost performance of a server, we need just both. It's hardware and it's software components to make it operate efficiently. When we talk about server optimization, we talk about optimization, and optimization can include fine tuning of web servers, we can use Apache web server, we can use engine X web server, we can use live HTTP servers and so on. We can talk about disk optimization, about block devices, about rate, about different file systems on server, including SCSI devices and SS drives devices. We can optimize our kernel on our server. We can optimize network, TCP stack, or TCP IP network stack. We can optimize firewall rules, or we can optimize some, just all of these things. When we talk about server optimization, we talk about databases optimization, we can use MySQL database, we can use Postgrade database, Oracle databases. So, then we must use benchmarking and profiling tools, and we must find bottlenecks, what stops our server. We must optimize settings in our server. We are talking about data storage, tuning, and disk and memory usage optimization. But where we must start? It's always start with installation of server. You must always make custom installation of server. Don't use default settings. Don't just click next, next, next and then finish. This is the worst thing, what you can make. Always do custom partitioning, because when you install default server, the partitioning scheme is like swap partition, root partition, and that's all. When you are doing custom partition, always make separate partition for directory or partition, TMP, boot, root, home, USR, OPT, and such directories, because if some of these partitions fill up the system, the Linux will stop working. If you have custom partitioning, if you have many partitions on your drive, you can choose between multiple file systems. Some file systems are better for some work like the other file systems. You can use EXT4 on root file system and you can use some other file system like EXEFS on the other file systems. Then install only needed packages. Never install whole group of packages. Make minimal installation and then add packages, which you need. Then you don't need Exvindo or GNOME on your server, or you maybe do, because if you make installation of Oracle Linux, Oracle servers with Oracle databases, then this installation is dependent of XOR packages and so on, but you can install only these packages and not the whole group of packages. And then don't use frame buffer devices on your server. Text console on Linux is 80.25 characters long and not this like SVGA resolution. When we are talking about performance monitoring, then we are talking about that Linux system administrator should be proficient in Linux performance monitoring and tuning. To identify system bottlenecks and come up with solution, to fix it, you should understand how various components of Linux work. Do anyone know where is the best documentation about Linux, about Linux hardware, the protocols, services and so on, where you can find it. You can find it in on every Linux distribution. Yes, main pages, kernel documentation, yes. The kernel documentation is the best thing where you can find everything about not about Linux, about computers, about network stacks, about disk drives, about kernel, about services, protocols and so on. This documentation is the best thing which Linux has. On a very high level, the following four subsystems needs to be monitored. The first subsystem needs to be monitored is CPU, the second is network, the third is IO and the fourth is memory. When we are talking about CPU, you should understand that the fourth critical performance matrix for CPU. For first, we are talking about context switch, then we are talking about run queue, then we are talking about CPU utilization and we are talking about load average. Everyone has heard about CPU utilization and load average and maybe not about context switch and run queue. What is context switch? When CPU switches from one process or treat to another, this is called context switch. When a process switch happens, kernel stores the current state of CPU in the memory. Kernel also retrieves the previously stored state of a process of treat from the memory and puts it in the CPU. Context switch is very essential for multitasking on the CPU. A higher level of context switching can cause performance issues. What is run queue? Run queue indicates the total number of active processes in the current queue for CPU. When CPU is ready to execute a process, it picks it up from the run queue based on the priority of the process. Process that are in slip state or IO weight state are not in the run queue. A higher number of processes in the run queue can cause performance issues. How do you know which processes is run queue or how much is your server loaded? You can know by CPU utilization. CPU utilization indicates how much of the CPU is currently getting used. Where can you find CPU utilization? We do each comment. The most common comment is top. Everyone knows top or comment like this is top. The essential Linux comment, you can try maybe A-top. I must be a root user, just a second. This is A-top. We don't see this first line, just a second. This is A-top, very similar to top. Maybe always more and more popular is H-top. With colors and these progress bars and so on. This is the common monitoring tools for CPU utilization. 100% CPU utilization means the system is fully loaded. When your system has 100 CPU utilization, when the load number, the first number is how much? One per core. Yes, thank you. And this is this line. A half percent of CPU utilization will cause performance issue. What is load average? Load average indicates that the average CPU load over a specific time period. On Linux load average is displayed for the last one minute, last five minutes and last 15 minutes. We can see this here. One minute, last five minutes and last 15 minutes. And one example, load average of 0.25, 1.20 and 1.90 indicates that the load on this system is coming down. Why? Because 0.25 is the load average in the last one minute. 1.20 is the load average in the last five minutes and 1.90 is the load average in the last 15 minutes. And this is now load average of this netbook in the last one minute. If I run top, I can see that the process is ex-vendor on my system and if I run maybe something like Firefox, just a second Firefox, it will load some pages, maybe one page with flash or something like that. I hope it will work. Yes, this is our home page from the company and I turn off Firefox. Take a look at the load and the load is a little bit higher. But if I run top in the middle of that Firefox is running, the load will be about 1.0, but netbook is not so much good computer. This load average is calculated by combining both the total number of process in the queue and the total number of the process in the uninterruptible task status. Then I have said that I will say something about disk IO optimization. Linux currently ships with four different IO schedulers. Did anyone hear about IO schedulers? Do you know what it is? Okay, someone else. And do you change these schedulers or just put the default values? Ah, okay. I have one line where I talk about SSDs that maybe it will help you. These schedulers are Deadline, Noop, Anticipatory and CFQ. There are many differences between these scheduling algorithms. CFQs algorithm is the default algorithm in most Linux distribution. It attempts to distribute all IO bandwidth evenly among all processes requesting IO. It's ideal for most proposals. When you check which scheduler do you use, you will see that you are using CFQ. Noop algorithm attempts to use as little CPU as possible. It acts as a classic FIFO, first in first out queue, expecting the hardware controllers to handle the performance operations of the request. Then Anticipatory, this algorithm attempts to reorder all disk IO operations to optimize disk 6. It is designated to increase performance of the systems that have slow disk. And the fourth deadline, this scheduling algorithm places IO requests in a priority queue, so it is guaranteed to be run with a certain time. It's often used in a real-time operation system. If you are building a real-time operating system, real-time kernel, then you must use the deadline algorithm because it's the best on these systems. How to know which algorithm do you use? This is, it's saved in a sys file system. Just cat slash sys slash block slash sda, sda is the name of the disk, queue scheduler and IO scheduler, which is in this bracket CFQ. This is the default algorithm in Linux and I have here default installation. Next week, I will put SSD, solid state drive disk in this network and I will see if anything will be better. How we change algorithm without any problems? Eho, noop to this scheduler and our algorithm will be noop. If you use SUSE Linux or open SUSE, you can change this algorithm via JAST, yet another setup tool. Maybe more about system scheduler. Changing schedulers on the fly allows you to test and benchmark the algorithm for your specific applications. You can change this with this common on the fly and test applications. Again, you don't need to reboot your server or something, some other things. Once the changes issued, any current I operations will be executed before the new scheduler goes into effect. So the change will not be instant. You must wait a few seconds, maybe a minute. Also remember that once one is set and performance to your liking, be sure to set the change to be applied on subsequent reboots. You can must change this with this CTL comment. It's often recommended to use noop or deadline on SSD drives. I don't know that, I must check this. This is like they said, it's good, but really I haven't tried it. There is usually no definitive answer to which algorithm to use. You must benchmark each one to get the best options. There are cases when CFQ may not be the best scheduler for your system or applications. As an example, if you are running a RAID disk with caching RAID controller, in our company we are using this default CFQ, but now on few newer systems which we have installed in the last two, three, four months, we are using deadline because we have recompiled our kernel, and now we are using real-time kernel for our software. IO optimization. IO wait is the amount of time CPU is waiting for IO. If you see consistent high IO on your system, it indicates that problem is in the disk subsystem. You should always also monitor read seconds and write seconds. This is measured in blocks, in example number of blocks read write per second. Terral also referred us by and bow, block in and block out. You know which is the common for measuring IO optimization for measuring IO stat. TPS indicates total transaction per second, which is summary of our TPS, retransaction per second, and VTPS writes transaction per second. I hope I don't have it installed. I will install it later. Then when we are talking about disk IO optimization, we must choose which file system we will use. The available file system is on the most linuxes are AXT2, AXT3. Yes, its Razer FS is still available on SUSE Linux, then it's AXT4 in butter FS. It's always more and more popular. Then when we are talking about disk IO optimization, we must use or change options in FS tab, file system tab. This file and these options. If we want much more higher speed for access to some files, it's good to use options no time or no dera time. That access time will be not written on the file. If you choose, if you use some, yes? Sorry, sorry. Yes, yes, thanks. Thank you. When you are using networking file system like NFS, networking file system or CIFS, common internet file system Samba SMB, it's a really good choice to use this stuff. The access to your files will be much more higher. Then when we are talking about disk IO optimization, we must choose if we have hardware rate in our server or software rate. We can choose between rate 0, 1, 5 and 10. Write 5, it's not a good recommendation on linux systems because if one disk is in failure state or broken, your system will be very, very slow until you change the disk. For disk optimization, it's always good to check benchmark tools like Bony++. It's really good and very old benchmark checking tools. It's available for every linux distribution. You can find it also on live CDs. You can use HDPARM, HDPARM is for older disk and SDPARM. First thing, when you buy new server, you must always upgrade bios of your server and firmware of your disk. The latest thing I have read about that a few days ago, is disk park hat. Some disk SAS or SATA disk park hat. If your disk park hat, it will be slower when you need access to the files. With new firmware for the disk, you can choose options that your disk will never park hat. The next is network tuning. For network tuning we must have a good understanding of TCP. Ip concepts and this is very helpful where we are analyzing any network issues. For network interfaces, you should monitor total number of packages and bytes received or sent through the interface. Number of packages dropped, collisions, et cetera. When we are talking about TCP tuning, for servers that are serving huge numbers of concurrent sessions, there are some TCP options that should probably be enabled. With a large number of clients doing the best to kill the server, it's probably not uncommon for the server to have 20,000 or more open sockets. If you want to change these things, you must change some default settings in the proc file systems via ccl.conf file. If you need to allow more local ports to be available, you just put this command. You can then increase the amount of memory associated with socket buffers and that you can often improve performance of your network. Then you can reduce the amount of the work that TCP stack has to do. It's often helpful in these situations to turn off these settings. These slides are also available on Linux foundation page that you can try these. You don't need to write down these comments. When we are talking about TCP tuning and you are having 10-gig network interface controller, it's always very good to increase TCP maximum buffer size setable using with setsock options. These are comments for ccl.conf. Then increase Linux auto tuning TCP buffer limit. Then increase the length of the processor input queue. Then to recommend default congestion control is HTTP and recommended for host with jumbo frames enabled. If you have such higher speed network interface controller, it's always very best that you try to enable jumbo frames. Then TCP congestion avoidance algorithms. Every Linux distribution uses very old TCP congestion avoidance algorithm and this old algorithm is Reno. Reno is traditional TCP congestion avoidance algorithm used by almost all operating system. It's default in all operating system. The best in Linux is that you can change this algorithm without any problems. Then the second algorithm is cubic with the name cubic TCP. Then BIC Algorithm HTCP. This is this algorithm here. Then is Vegas. Maybe you have heard about Vegas TCP Vegas. It was very popular and it was developed at the end of the 90s. Then Westwood Algorithm which is optimized for losing networks. In Linux and Unix operating system you can change it without any problems. In other operating system I think that you cannot change it and you must always use Reno till the next addition of your system. How do you determine which do you use? You can try with this command cctl TCP available congestion control. Let's try it. This is this. This congestion control is cubic and Reno. I used the first one, cubic. Cubic is now default in newer Linux distributions on this network. I am using Debian 7. Then memory optimization. When we are talking about memory. If you have 16 gig of memory installed on your system you have 16 gig of physical memory. Virtual memory is a swap space available on the disk without physical memory. The virtual memory contains both user space and kernel space. Using either 32 bit or 64 bit systems make a big difference on how much memory a process can utilize. On a 32 bit system a process can only access a maximum of 4 gig virtual memory. This is the common stuff. You can see this on Windows operating system on Linux where you get 4 gig of memory on 32 bit system. On a 64 bit system there is no such limitation. This limitation with 4 gig of virtual memory is also problem with swap space. If you have 32 bit operating system and you need more like 4 gig of swap, space swap partition always make 2 4 gig swap partition. Never make 1 8 gig partition because 8 gig partition will not work on 32 bit systems. Unused ROM will be used as file system cache by the kernel. Linux system will swap when it needs more memory but disk is always lower like memory. When it needs more memory then the physical memory. When it swaps it writes the least used memory pages from the physical memory to the swap space on the disk. This is very slow. Lots of swapping can cause performance issues as the disk is much slower than the physical memory and it takes time to swap the memory pages from memory read access memory to the disk. When we are talking about memory optimization on a newer system we are talking about the dense memory. This is hardware specific and it's not Linux specific. Then we are talking about Numa. Numa is non-uniform memory access. Then we are talking about huge pages. About huge pages you have heard about that. Then we are talking about managed virtual memory pages. What is Numa? On newer system we have one demon with the name Numa. Numa exist from Fedora 17, now we have Fedora 19 and on Red Hat Enterprise Linux 6.3 as a technical preview. Numa is a user-label demon which automatically improves out of the box the Numa system performance. It's not enabled by default. It monitors available system resources on a per node basis and assigns significant consumer process to align resources for optimum Numa performance. Numa demon or service rebalances when necessary and provides pre-placement advice for the best initial process placement and resource affinity. If you need Numa on your servers just and if you use Fedora 17, 18, 19 or maybe Red Hat Enterprise Linux now 6.4 you can turn on Numa because it's not enabled by default. Then huge pages. Two Mac pages are versus four kilo pages standard Linux page. Virtual to physical page map is 512 times smaller. TLB can map more physical pages, but I think in a fewer misses. Traditional huge pages are always pinned and transparent huge pages does exist in Red Hat Enterprise Linux 6 and in all new version of operating systems, not operating system, new version of distribution. Most databases support huge pages. If you use Oracle databases you must turn on huge pages and one gig pages supported on a newer hardware. How to configure huge pages 16 gig? You can just change these values in prog file system or change this CTL.conf number. Then flashing cache. You can drop unused cache with this U3 unused memory file cache in the databases uses cache may notice slow down. If you need to free page cache just execute this command. Wait maybe 20, 30 seconds, maybe one minute and this cache will be flashed. Then if you want to free slab cache just execute this command and if you want to free flash page cache and slab cache just echo 3 to proxist VM drop caches. You can try this on your servers where you run databases. Then a few words about Swapiness. Swapiness controls how aggressively the system reclaims mapped memory. Default is 60%. Then we decrease then more aggressively reclaiming of unmapped page cache memory. When we increase this number then we more aggressive swapping of mapped memory. How to change Swapiness? Just a second. This is the default value. When you are using server system it's recommended to decrease this number. On your desktop systems are like your client laptops is advice to increase this number. When you are making optimization you must always remember the 80-20 rule. You know maybe 80-20 rule. It's one Italian rule that says that just a second I have copied this. Wilfredo Pareto, who observed that 80% of income in Italy was received by 20% of the Italian population. And this rule is now in everyday life. If we translate this to our computer IT language then 80% of the performance improvement comes from tuning the application and the rest 20% comes from tuning the infrastructure components. When we are talking about system monitoring tools, the most common monitoring tools which is installed on every Linux and Unix system is VmStat. For virtual memory statistic, netstat, for network statistic. We all know how does netstat look like. Which ports are open on our server, on which protocol, what is received queue and what is sent queue. My received and sent queue is empty if you see here very large numbers then something is wrong with your network. Then you maybe have a lot of dropped packages, maybe some collisions and how you determine if there is any collisions on your system. I have config here. Received package errors, dropped, overruns, frames, carriers, overruns, dropped errors and this is the same for received and transmit. Then PS process status, you all know this comment. Then top, H-top, M-top, iostat and maybe some graphic color representation of this comment XOS view. But you don't have installed, like load, CPU level, CPU 1 level, memory disk, viral slangswap, memory pages and so on. It's a very old application for every Linux system. Then when we are talking about kernel tuning, yes, the best solution is to recompile your kernel. But if you have some enterprise distribution like Red Hat Enterprise Linux or Susel Linux Enterprise Server, then you will lose your support. If you are making recompile of your kernels then you must exclude unneeded modules because there are many unneeded modules in kernel. You can use real-time kernel but you don't need to buy real-time kernel. You can build real-time kernel from your existing kernel and you can make kernel smaller. You can make kernel smaller with excluding unneeded modules. On the internet I have found some recommendation which thinks we can turn off in our kernel. If you send me an email, I can send you this table. This is a table of a kernel configuration options, including a description. The default values for a kernel and the recommended value for a smaller kernel, depending on what you need. This is the kernel configuration options, description, default value. If you want a small kernel, you must use this value. If you see here, yes, it means that you can use this but with caution. If you see here a number, you can just change this number. This is a table with which you can make your kernel smaller. Kernel in our systems are big. This is our kernel, vmlinus, l minus h. It's 2.6 mega. Yes, it's not big but this is Debian distribution. If you can take a look at Red Hat Enterprise Linux or Social Linux Enterprise Server, it's about 4.6 or 4.9 meg. In these are only kernel and static modules. Then the other modules in your kernel is saved in lib modules 3.2686. 83 meg of other modules, which must load on system start and so on. This kernel is not so big but because it's compared to real-time kernel. If you take a look on enterprise distribution, how much space does this directory take? I think about 500 maybe more meg. When we are talking about samba tuning, we can enable IAO but this is the extreme options. Then we must change socket options. They say this is the best socket options for samba tuning. Then we can enable raw read and write. We can use opportunistic locking. If we use samba, does anyone use samba? If you use samba, your log level is set to maybe number 3 or 4. Use the smallest possible log level because if samba writes logs to your hard disk, it will slow down your server. Then about database optimization, yes, the most popular database is maybe MySQL. They provide a lot of scripts for MySQL tuning. There is one per script for MySQL tuning application. You can optimize tables, you can use INODB or MyISOM tables. You can try some other databases. When we are talking about open LDAP tuning, we have a big LDAP tree in our organization. You can index all the things in this open LDAP. If you add the following parameters to slap.conf before entering the info into the database, they will all get index and performance will increase. But you must do this at the start. If you use Apache, you can start a massive amount of HTTP process. But if you start a massive amount, it's a really benchmark hack. You can change these values without any problems. Or you can use optimizers, memcache. You must tune your Apache with this. You must minimize the number of Apache modules. In Apache, they have a lot of modules which you never used. Or maybe try some other web servers, change maybe Apache of engines depending on what you need. Then what is benchmarking? A good set of benchmarking utilities are often very helpful in doing system tuning work. It is impossible to duplicate the real-world scenarios on your test. Or test environment, but that isn't really the goal of a good benchmark. A good benchmark typically tries to measure the performance of one particular thing very accurately. If you understand what the benchmarks are doing, they can be very useful tools. The most popular benchmark tool for this benchmark is Bony++. D-bench is a tool to generate IO workload to either a file system or to a networked NFS server to come an internet file system like Samba. Then we know HTTP load, which runs multiple HTTP fetches in a parallel to test the throughput of our web server. Then DKFTP bench, the same, but for the FTP protocol. PIO bench is a multi-treated IO benchmark tester. TTCP is a utility for measuring network throughput and maybe network performance tester. Then for the end, when you must identify and solve performance issues, you must understand what is the problem. Half of the problem is solved when you clearly understand what the problem is. You must monitor and collect your data. After defining the problem clearly, monitor the system and try to collect as much data as possible on various subsystems, you must eliminate and narrow down issues. After having a list of potential issues, you must dive into each one of them and eliminate any non-issues. And last but most important, make one change at the time. Don't try to make multiple changes at the one time, because you will never know what is the help for your system. I think we have talked about 50 minutes. That's all. If anyone have any question, you can also write me mail. This is my private email address, company webpage. My blog is in Slovenian and English language and my CV or personal webpage with links to my social networks. Does anyone have any question? Yes? Can we find this presentation online? On this program of this conference, there is link to presentation. So just click on the speaker named David Gutierrez and there we will link. If you won't find the link, just write me mail and I will send you. No problem. Anyone else? How? Where? The current documentation is in USRC-Linux-Documentation. This is one big, really big directory with subdirectories, network, disks and so on. It's very helpful. Bless you. Anyone else? Okay. Thank you for hearing.