 Hello, I'm Jaroslav Škarvada, software developer at Redhead. And today I would like to talk about Q&A project and how it could help you with system tuning. Maybe you have already had some presentation mentioning Q&A or maybe you have already seen it in some performance tuning guides, but these papers usually do not focus on tuning in detail, so that's why I created this presentation to introduce the tool more. Regarding system tuning, if you are a system administrator, usually you don't are okay with the tunings, with the default settings that are in a Linux kernel and Linux system because you want to get the most from your expensive hardware, so that's why usually you need to customize some tuning knobs in the system to get the best performance. Also it depends what performance you need. For example, if you are running some high-performance computing cluster, you want to tune your system for the highest possible chip pad. So to be able to calculate more numbers on your system and in case you are running some message browser, you need to tune your machine probably for low latency. If you have some backup cluster, you probably may want to tune it to consume low power. So usually if you need to do some tuning, probably people consult guides and best practices forums and knowledge-based articles and there are some bunch of commands that they usually just copy and paste, or if they want to repeat these, they usually put it into scripts. So it can very, very soon get very complex and hard to maintain because there are many knobs on the typical system. You can tune. For example, cctl, ccfs, various tools. You need to call to do the tuning, the tools at configuration, usually in ETC. Also you may need to boot kernel with special parameters, run or tune system-d services and also there is, you may need hotplug. For example, if you attach new block devices to this system, you may need to tune it according to your needs. For example, change the IOU schedule or similar. So there are some more problems you can encounter with this approach. But this approach usually works because it's used for decays. And the other problems are the maintainability. It's especially a problem if you need to support multiple versions of operating systems, multiple versions of kernels because even the kernel interfaces and API are stable. They can evolve over time and then there can be changes and non-compatibilities. So usually you end up with several scripts for several machines in your network. If you have several versions operating systems, or you can do one very complex script that will handle it all. And the problem is that you can very easily get lost in it because you can just check some line and you can't remember why it is there. The other problem is how to verify that your system is tuning the way you already want. So usually you can't say that everything works as you wanted. So you probably need to run some benchmark to verify that the system behaves the way you want it. And there can be also interference with other tools because maybe you tune your ccpl somehow and then some other software is run and it changes the ccpl settings. And you will not notice and your system will be tuned differently than you wanted. And last but not least, the rollback. Usually with the script approaches you don't take it into account and if you need to go back with these settings you just provision the system, reinstall or temporary settings just reboot. So Tundi is a small project that tries to address this and help you with the system tuning. It has a plugin architecture, so it can be easily extended. Tundi is centralized in one place in so-called Tundi profiles. So you don't need to take care of all the settings, you just need to know what Tundi profiles to use. Rollback is supported, hotpack is supported, verification is supported and also Tundi can do some auto configuration of itself. It can detect the hardware or operating system versions and type and then can auto select the profile according to this detection. The tool can be controlled from a command line interface or through debas, so it can be easily integrated to existing solutions. For example, it's supported in cockpit. We have also an unseable role for it and the great thing is that it's available in Red Hat Enterprise Linux. In Red Hat Enterprise Linux 7 is already installed and enabled. So this is probably the most important slide for you if you are just ordinary user. People usually ask us for some magic button to just push and don't care about the details. So maybe the Tundi ADM profile command is near to it because all what you need is just to type Tundi ADM profile and then the profile you need and the profile will be loaded and all the system knobs will be set according to this profile. If Tundi is not running on your system, if you don't have Red Hat Enterprise Linux 7 or it's not enabled, like in Fedora, you also need to start it through system CTL or you can also enable it to auto start. The other important command is Tundi ADM list, which will show you all available profiles on your system that you can use that are installed and available. Tundi ADM active shows which profile is loaded and finally there is Tundi ADM verify command that just checks whether all the settings from the profile are applied. If there is something different, for example some pool change, some cctl knob or something, it will show that there is a difference and it will show what is different and you can use this command, for example, for scripts just to check the dash i parameter is there to ignore hardware, to ignore differences that are caused by a non-compatible hardware or that you have hardware that does not support the tuning. For example, if in the profile requested CPU Freq governor to be set to something that is not supported on your system or CPU Freq is not at all supported on your system then Tundi ADM verify will exit error but if you use dash i parameter it will ignore it because it knows that it's caused by not supported feature in the hardware or software. There is also a hack-ish example on this slide how to control it through debas using the debas sent debug command and this do the same as the Tundi ADM active in this example. So regarding the profiles, we have several profiles, I'm stream and these profiles have settings that are provided to us by performance engineers so the random benchmarks do the tuning know what the customers need and these tunings from performance tuning guides are put into these profiles. There are profiles for general goals like throughput performance or latency performance or latency power saving and balance in between. We also have profiles for various products for example for SAP, for database servers, for Atomic and also there exist some head-party packages providing Tundi profiles. Regarding profiles for products they are made from the documentation from the products and from knowledge bases and best practices articles. So exactly if you don't need to go to the knowledge base and tune it by hand you just select the profile and everything is set up. If you want to roll back and try different tuning you can do it, Tundi can roll back the settings so this is also good for benchmarking so you can just tune the Atomic profile change profile and benchmark or system with different settings and you don't need to reboot or reprovision. Regarding reprofiles there are two types of profiles factory or so-called system profiles and custom or user profiles. The system profiles are under user lip and usually there is a problem that some people directly edit them. Please don't do this because if you do it every update of the system profile by RPM, by new version of RPM will override your settings. So if you need to customize it the better way is to copy it to ATC Tundi which takes precedence when selecting the profile and then do your changes. I will talk about it more later. So what's inside the profile? The profile is directory that could be different files used by different plugins and the main configuration file of the profile is called Tundi corner file. If we look more deep on it it's just any file that has some main section and in the main section there can be summary and description what the profile did. This is what you see in, for example, cockpit. You will see that list of profiles and description what they are good for. All this is optional. You don't need to have it there. Even the main section is optional. And then there are plugins configuration. So here in this example I have enabled this plugin and CCTL plugin. In CCTL this is the same just for setting CCTLS. So I am setting serpiners to 5 which is quite low. It means that the system will not swap if not necessary. Regarding the disk plugin there is set really ahead to 4096 and each plugin has a bunch of these options supported. In the form written there it will take all the disk devices or block devices non-removable on your system and tune it this way. If you need some more complex settings like tune different disk differently there is more for both form how to write it and these two are the same. So if you write more for both form you need to specify that it is a plugin type disk. This is the instance name it could be any arbitrary string but must be unique to describe your instance and there are devices the plugin will handle so it will handle all devices. On this slide you can see how to configure it for more instances. So this is the more complex form and you can see I have three instances of plugin disk. One I named disk system we are disk beta and the rest for the rest of disk on my system. So all are type of disk and I want my system disk I know it's SDA1 so I want SDA1 to be tuned this way SDA2 to be tuned differently. Here you can also see something called conditional change operator. This can be read as if just I want read ahead to be at least this number and read ahead is lower it's set to this number it's higher, it's not touched. So this way you can configure different devices. Also it's possible to do full udef regular expression matching. On this slide there are set two instances of the plugin to handle all these manufactured by Samsung so this udef property matched by regular expression and this second instance will handle all SSDs also it takes these devices and in udef it can be done by just checking that the rotation speed is zero. All properties you can match can be get from the udef by disk command, udef idm info udef property and the device you are interested in and all the properties written by disk command can be matched by sorry sorry so you can see that in the first example I am also setting I will schedule it to be deadline and the great thing about it is that it will not when this profile is applied it will not only tune your system all disks on your system but it will also tune later any devices so if you put in new Samsung disk at runtime and it will match any of this rule it will be tuned accordingly so there is another concept in the override which means you don't need to write the world profile, you can just base your profile on any existing profile so here you include the throughput performance profile it will take all the settings from it and you will just add the governor in the CPU plugin to be user space this syntax will disable all tuning from throughput performance for all disks and this will cause that the ccpl set in the throughput performance profile will be cleared and just this one will be so in the first form it will use anything and just modify what's written here and this is good for things called profiles chaining this is an example from the rpm package kundi profiles nfd which provide these profiles so it starts with the latency performance profile which tune your system for low latency then on top of it is based network latency profile that adds just tuning of network interfaces then the real time profile is based on it which adds tuning for real time kernel and finally the real time virtual host virtual host profiles so the good thing is that different teams can work and maintain these profiles and just focus on what's important for them and don't need to cope with the other details now how to customize the system profiles as I said you can just copy and update it but the problem is that you get just a snapshot of the profile provided in one time by the rpm package if maybe you are ok with it maybe you aren't so if you want the latest settings for the system profile to be in your customized profile you can use the overwrite feature this means just create new profile and include the profile you want to customize and customize it by overwrites so this is preferred by because you don't close your settings and also you will get the latest tuning from the rpm packages now I would like to show what we are also trying to solve in tundi to provide some abstraction to the changing interfaces so for example if you need to set transparency you can do it just by shell command but the command is different on different versions of kernel the path is this and on the seventh the path is this so if you need to copy in script you need to have some check there or two different scripts with tundi you just write this and it will take care about it automatically the tool is there for nearly 10 years so we are following the changes in these settings which are most needed and can provide abstraction layer for it there is another example for example if you need to change boot line boot kernel parameters if you do it by hand you will need to check whether it is bootloader specification enabled system in such case you need to call groupenable to edit the groupenable patch some entries check which bootloader is there whether it is grab2 in case it is not bootloader specification patch different files check whether it is EFI it can be quite complex with tundi you do not care you just write this it will set kernel boot command line out of the there iso CPUs we will set them to one which means that the kernel schedule is set to CPU por one so it is extended to more complex examples another thing in this profile is variables you do not need to customize the profile for example if the user needs to isolate kursi. Vse možete predstaviti stetik profil in drugi profil in počke počke. Vse možete predstaviti stetik profil in počke. Vse možete predstaviti stetik profil. Vse možete predstaviti stetik profil in počke. Ako počke počke počke počke počke počke počke počke počke počke. nr. one, this will return cores 2, 3, 4. And there are more functions supported in Tundi out of the box, and it's plugin-able, so you can also write your own plug-ins. For example, you can call external helpers, for example, to calculate parameters at runtime in the profiles applied. Another feature of Tundi is dynamic tuning. It's there to, initially, it goes at it to cope with things which wasn't supported in kernel. So, for example, if there is something, governance or policy doesn't exist yet in kernel, this can be done from user space. So, in Tundi, some plug-ins can do dynamic tuning. This is done, there are some performance counters, like CPU load, disk load, network load, which are provided by monitor plug-ins, so you can add other. And then the Tundi plug-ins can change these some settings in runtime according to these counters. This was experimental feature, and it's actually visible in the enterprise Linux, because we want to create two performance there. So, this will, this can change the tunings according to external factors. But if you want to play with it, you can enable it. You can be done in the minetune re-configuration files. If Fidora is actually enabled. Regarding the auto-configuration, Tundi can also check your system and preset, initially preset profile that it thinks is most appropriate for your system. You can do it by calling Tundi IDM auto-profile, or if you just want to check what Tundi thinks is the best on your system, you can call Tundi IDM and command. So, for example, if you have some product that needs some specific tuning, you can do Tundi profile to tune it, and provide the recommend file to automatically set your profile. Here is example of rules, which can be used. For example, this will set super performance on systems that have compute node, or server, in ETC system release, and are running on bare metal. So, no virtualization. This is output from the whipboard tool, and there are more checks supported. You can check for running processes. So, for example, if you find running debase server, it can assume that it's debase washing used for debase server, and can preset debase profile. So, if you have some rules, you can grant ETC Tundi recommend D, and it will be taken into account by the engine. Also, there is a feature called noDemon mode, because actually Tundi is a demon that listens on debase, and it can be a problem on embedded resources system, or some people just don't like another demon running on the system. So, you can set Tundi to run without demon by setting demon zero. And then Tundi will behave, just one shot. It will start, do its job, apply Tundi, then exist. Problem is, actually, it has less functionality in this mode. There is no debase control yet, but it's solable by socket activation. No clock support yet. It could be helped to do that. Just nobody write the support yet. No tuning of newly created processes. This is probably not solable in noDemon. Of course, no dynamic tuning, and no rollback. It shouldn't be problem to implement. So, regarding the improvements, we need better documentation, especially some reference manual, auto-degenerated, with the... All plugins and described options and what the options mean. Because at the moment, there are some documentation online, but it's not complete. It's a bit outdated. And if you are a new user of Tundi, it is probably not that easy to find out all this information. Also, we would like to improve the noDemon mode, support more functions. And we was asked to add so-called simulation mode to just display what will be changed by the profile and don't actually write the changes. Also, there is a space for improvement regarding containers and much, much more. So, a conclusion. If you don't use Tundi before, give it a try. It's enabled and installed in Red Enterprise Linux. It's slowly finding its way to other distributions. And if your project needs specific tuning and you are doing it by some wild script, consider using Tundi. We, as upstream, can merge your profile, can maintain it to run with latest Tundi Core, Tundi Engine. And you can just benefit from the infrastructure which is supported by Red Hat. Also, if you find back or have some future, don't hesitate to open back report or contact us. Because there are some people saying, hey, this project looks interesting. We might use it in our use cases, but we found out that there aren't features we need, so we will not use it. Just in these cases, open backzilla, we are trying to implement the functionality if it's possible. So here you can see some context for the project. And, well, we have some time left. So I can show you another thing which could be useful for desktop users. Tundi has cover-top integration. If you know cover-top tool, it's a tool written and maintained by Intel. And as the name suggests, it shows you the processes that consume most power. Well, normally it shows you processes that do most wake-ups of T-class. But if you calibrate the tool, this is a thing that not everybody knows by running power-top dash-dash calibrate and keep it running for probably one hour, it will then show estimation for each process how much power it consumes in watts. And there is also a feature in power-top that can tell you what you could do for improvements and lowering power consumption of your system. There are a lot of requests to power-top upstream to add some auto-configuration to this tool for people just hitting the magic button and power-top configuring with everything for them. But the power-top upstream insisted a lot. But finally, the feature got there, and there is power-top dash-dash autotune feature that will enable all tunings. But it can cause problems on various systems because there can be incompatibilities like your mouse can die if it doesn't support power savings correctly and others. So, there is maybe a better way by using power-top to tune the tool. So, just install tune the tools and power-top. And by running this tool, it will generate new tune the profile that will contain all the power-top suggestions turned into tune the profile. And you can just edit this file and select what you really need or not by commenting or uncommenting it. And finally, by activating the profile, you will get all the power-top suggestions on your system. So, this is how to integrate power-top with tune the. So, I have some other slides about some tune the functionalities like PM quality of service and real-time tuning of processes. But these are more complex examples. So, you can see it in this presentation if you download it from the default site. So, probably, this is everything from me. And thank you for the attention. And if there are any questions, I am here to report them. Yes? So, you briefly mentioned the AC capability of tuning to be dynamic in frequency range system. Where does it take the input from, in that case, from the system, but how? You mean the dynamic tuning functionality. The question was from where it is taking the information for dynamic tuning. It is taken from the monitors plug-in, you can write. And currently, there are plug-ins to collect CPU utilization or CPUs utilization, network load for your network card interfaces. And also, you can write probably any monitor you need to fit your net well. The question was whether it will be possible to write some function to fit the TCP data. PCP data to, through the plug-in. Well, I think it could, the plug-ins are written in Python, so you can definitely write everything you need. Any other question?