 Yeah, no, so this should be running on your host as it is, yeah, as a part of it, yeah, installs the VM. So I'll try it, so let's do this, I'll just show it on my part because we were supposed to distribute these USB sticks and they are not available now. So I'll go ahead and do the workshop myself and if you want you guys can carry along like later on after the workshop. We need a VM basically, no other option. So if your pip is failing you have to like equals equals 0.7, let's do one thing, I'll come to this part later on and let's get started with the workshop now. So hi everyone, I'm Arshad Sharma, I'm from Red Hat Performance Team in Bangalore and I'm going to talk about analyzing latency of IO events here and so things we're going to talk about. An IO use case we'll pick up and so how we got to this point was we were investigating latency in the KVM QEMU layer, QEMU KVM and so we were doing some block IO tests and so we came down to comparing native and thread setup so we released a couple of tools as a part of that and that's what I'm here to demonstrate. So we have also released, so I'm particularly in the tools team there, so I developed tools for the performance team and so we have released this benchmarking platform as well which is called P-bench, I'll come to this later in the slide and we wanted to demonstrate that as well so it can help you out analyze latency, run benchmarks for debugging, tracing system calls and stuff so that you can improve your performance or tune your system accordingly. So the use case here is that IO events in QEMU KVM, whether the delay is produced by the file system or the KVM layer, whether how does async compare to async or how does a setup with target threads compare to target native would achieve better results if I change the IO depth, how does block IO and file IO compare in a particular setup. So let's talk about block IO events in QEMU KVM, so by latency analysis here I mean we have event loops, when you run this tool called Power Script you'll have your events coming in, recording in, for example you shoot up a VM virtual machine and so the way it talks to the system is how for example you have some workloads that you run on the VM and the system calls that go on in the backend, so they have some event names associated with them and these event names have some, so if you want to see what's the submission time of a request or what's the completion time then you can trace those particular events and get the time stamps and see how much latency is, like how much time is being spent in form from, so you guys know about KVM exit, KVM entry in these events, everyone? Yeah, so for example if you have a KVM exit and you have an IO submit, then how much time is being spent in between that, so this tool is nothing but the Power Script Post Processor, the tool that I have released, it's nothing but it just calculates the deltas as in the display differences between the timestamps and gives you a visualized idea of what's being, like how much time is being spent in there, so for example KVM exit to sys exit people, then to sys entire IO submit, then to exit IO submit and so on to IO get events and so on. So this tool gives you a really, so whatever general approaches we have till date, we have this FIO benchmarking tool and we have an add-on from the tool that we said P bench, the benchmarking platform, we have a P bench FIO script in there, so P bench is, I'll come to P bench later on, for debugging we have, we have, everybody uses Perf tools and what I read on was this latency Post Processor and so yeah, let's talk about P bench a bit, so P bench and Post Processor they were released around 2015-14 and yeah, so P bench is this benchmarking platform which is like supports all your commonly used benchmarks as well as you can introduce your own scripts, custom benchmarking scripts, you can use them, you can run your benchmarks and you can get the visualizations as well along with the standard division, your different sample runs and so it really eases out the process of doing a benchmarking and it has got these three particular components, a collection agent which will handle all your telemetry, logs and configuration, so you specify whatever, for example you have different MP stat, IO stat, all these tools you can specify which are tools to run and they will collect your system data and then there's this background tasks which are basically three components as in there, you have a server where you're going to post your results later on after the completion of the benchmark and then you have this client machine where you're going to run the benchmarks, so this tool runs the benchmarks, collects them and displays the visualizations on the remote server, something like this, so for example you have, this is graph from IO stat, so you can have all your volume groups or block devices listed down there and you can have those visualizations right here according to the timestamps and this can help you with the various bottlenecks as in why the system was performing, for example low or higher than the required state, then we have this tool, the per script post system which I just told you about which basically analyzes those event loops that you have, so event loops you can, it's a flexible tool, so you can specify your own events, so for me I entered the virtualization field really recently I was helping out this guy from our team on the virtualization to perform some benchmarks and latency analysis, so I don't know a lot about event loops but if you know about the event loops then you can specify your own event loops, for example instead of KVM exit you want to see some other system call and you want to see the timestamp differences there and then after getting the deltas if you see that this particular delta has a higher, it's taking a lot of time then you can go ahead and look at the traceback from the system and see if what our system calls are being done, for example you're comparing XFS and you're comparing XFS inside a virtual machine as compared to a bare metal then you can see like if it's a problem with the XFS layer or with the KVM layer and so this is available on PIP and you can scroll through that and what it does is basically if you run your perf tools you'll get this perf.data binary file, you process that and then you get the mean, median, standard division of the event loop latency, that's as simple as it gets then it visualizes those, for example delta sys exit, six exit people to KVM exit or delta sys enter IO submit, I mean what's the difference between the exit people and the enter IO submit and all, so you can look at for example in this one we have a lot of difference between like the maximum and the minimum and you can see why so and so time I mean system calls are being taking a lot of time so you can look at that it's just a tool to help you out further your progress or progress your investigation for virtualization so there are some additional utilities as well that we have released so if you want to get started with the analysis of latency then you can run this tool called bench ita.sh it will use this P bench and it will run that and it will collect some data then give you some results with standard divisions and all so if your IOPS are low on a machine then you can use this to see where that bottleneck exists. Latency analyzer is the same thing so it's just a startup script you can run so I was hoping that you guys actually could do a hands-on experiment with me but we had this problem with the USB stick so you might not be able to run this right now so I'll just do a sample run and go ahead and let you take a look at what happens if you guys want to try it out though I have the VM image with me like as a QCao2 image and you can get it from pendrive do you want to like circle this pendrive around and copy this stuff there's this QCao2 image I mean instead of building it up on your computer you can anybody has any questions like what they're expecting and what they're seeing had something like that is like on first workshop so anything about latency analysis or b bench or first script processor I should probably give you some time to set up the tools on your computer so yeah yeah on the so when you run an FIO you'll supply a client as an IP or a whatever host name that you have so it will go ahead and run that on the VDB disk that it attaches to that VM and yeah so whatever events are being going on between the host and the guest it will all trace that you have trace your native trace points in the host and then you enable trace points to you yeah yeah I mean only guess what do you mean by enable enable trace points so it's not all okay yeah this is here that's what it does yeah so I just pushed this small change to the script which if your pip is failing on your computers then now that won't no hopefully please feel free to ask me any question I can come to a desk and sort it out if you don't if you are not able to install the tools please so then I think your pip is version 3 or 2 so then you can just remove the 2 and install that sorry so do you have this to cut to image right you have to cut to images in there just copy them over your computer in the and put them inside the wild images okay you have your own Fedora 22 yeah you have two images inside one and one and one yeah both things inside your wild images okay let me give you a little bit yeah wild live world images yeah copy them inside them basically I'm distributing pandas to copy this image to your computer there was a USB stick not available so it's such a drag I know it's going slow user in the past I've used the okay this so So, this latency analyzer script is doing nothing, but just installing your dependencies and bringing them the bringing of the VM and chatting it down for later purposes. We have a. We have a. Can you, I have an error. Okay, sure. So, I just pushed a change, you have a long list of things. I don't know. We got it. Please do a gate pull on the latency analyzer clone you did right now, if you haven't done it yet. So, I just pushed a change, you have a long list of things, you have a long list of things. So, I just pushed a change, you have a long list of things, you have a long list of things. So, I just pushed a change, you have a long list of things, you have a long list of things. So, I just pushed a change, you have a long list of things, you have a long list of things. So, I just pushed a change, you have a long list of things, you have a long list of things. So, I just pushed a change, you have a long list of things, you have a long list of things. So, I just pushed a change, you have a long list of things, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, you have a long list of things. So, you have a long list of things. So, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, you have a long list of things. So, you have a long list of things. So, I just pushed a change, you have a long list of things. So, you have a long list of things. So, I just pushed a change, you have a long list of things. So, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, you have a long list of things. So, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, you have a long list of things. So, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, you have a long list of things. So, I just pushed a change, you have a long list of things. So, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things. So, I just pushed a change, you have a long list of things.