 Our next speaker is Alastair Caden and he is going to talk about trekking local storage configuration on Linux. Please welcome Alastair Caden. Okay I'll go fairly quickly because we're running behind, right. So straight in. So is this yes this is working right. So common problems it's so my system has got problem with its storage devices is this shout if it's not coming through it doesn't sound right here. My system's got some problem it triggered thousands of log messages from different layers of the storage stack and applications on top so we've got this problem on Linux that we get lots and lots of error messages from different layers and they're all incompatible and very difficult to automatically process. So the challenge is to recognize all the messages that we're getting and correlates them and try to work out what the original causes that has triggered all these thousands of messages and in the storage area in particular we have a lot of problems when we reboot devices get different major and minor numbers but it might be the same disk so a disk might be having problems but in the log messages it appears to be a different disk every time after you've booted you get different numbers each time and similarly a lot of the information in Dev and Sys is transient connected with that it can change and we're not recording those changes properly so we're not keeping records and the basic challenge is can we keep better records of what's happening in terms of storage and having got those records can we analyze things automatically any better so the first thing we need is a persistent identifier for each device now this was solved a few years back when you'd have came along and you'd have rules were being invented and you see in the Dev disk directory lots of classifications by IDs labels paths UIDs and if you go there you can find your device and map it back to the major minor number and anything else relating to it even across boots and other changes on the system we want to record this information somewhere persistent so that we can query it later the system journal now part of system D can store structured data and it's as key value pairs it's no longer just a single message string like the original sys log used to do our sys log can also store key values but not to the same extent the system D journal D can do so having stored it we if we put it into the system journal the journal CTL command is capable of filtering this by identifiers but it's not a very sophisticated filter yet it just does fixed string matches at least wildcards and complicated patterns are beyond it at the moment but once we've recorded all this information going back to the time of the last boot we can play back the changes and from that reproduce the configuration like it was at an earlier time so what I'm leading towards here is if you're in a customer support situation you can contact you can get a copy of this from your customer and then replay the system to see what the storage was actually looking like at an earlier time perhaps at the time that you're trying to focus on and when you think the problem happened so storage logger is a very simple little package whenever the block device is added or removed it creates a U event this triggers what's called a Udev rule processing and this performs actions it finds out the type of device it probes it finds out what might be on it is there a file system there do I need to mount it and so on creates the nodes and symbolic links that you see in dev what storage log is going to do is record the result of all that into the system journal at the same time it can grab some information out of sys and attach it to that information so at the moment it is a bash script the script runs as a last Udev rule but in future if it successful what it does can be folded back into those earlier Udev rules and the remaining part could be built into Udev itself by either capacity to actually log in the system journal Udev could do that itself if it was worth doing. Secondly the replay part what I've done there is written a wrapper around LSBLK LSBLK is a very friendly tool that lets you see your stack of storage devices and find information out about them so I've written a pearl wrapper around that it takes time arguments since and until which happened to be the arguments that the journal CTL uses play back view events between those two times create temporary dev and sys directories similar to the original ones and then run LSBLK on those temporary directors instead of the real ones and people have suggested other things we can do with this we could take this sequence of U events that we've logged and the changes to dev and sys and do them as a series of git commits so you can move forwards and backwards in time using git that's another idea that's been suggested we may do finally there's a tool called skydive there've been some talks earlier today about this primarily this was invented for networking but it can also do storage so the thing about skydive is it gives you a GUI onto a graph database so we can store the relationships between the devices in the graph database we can also add in performance metrics I owe statistics into that same data brace and then use the skydive user interface to query that database and display it so I'm now going to move on to some specifics journal CTL these are some of the parameters all the messages I'm recording I'm tagging them with that particular string you dev log so this is the parameter to journal CTL to select those messages I'm asking for verbose output because I'm not actually logging any message string I don't want any of this to appear in normal VAR log message files it's just structured data so I'm asking here for verbose output I'm saying which fields I want I what keys persistent storage ID major and minor in my particular example and giving a range of times 10 minutes when I might have been giving the talk if we were running to time and finally the persistent storage ID acts as a filter to say only show me the events where the persistent storage ID is device mapper name volume group one logical volume naught that output can then be put through into LS BLK I put a J on the end because it's coming from the journal and it's also easy to type the until tells it when to stop so if I actually run that for real this is what I see so running that command on a virtual machine guest so third lined down you can see I'm look at the fedora KVM nor guest 10 route etc and this is the actual output you get the journal CTL log date and time stamp and then the parameters it had minor naught major 253 and these are the two strings that recorded as persistent IDs from you dev so these are the things you normally want to filter on so if I activated this device deactivated it did other things in between the major and minor numbers could be changing but if I filter it this way I can see all the events relating to that device and you can do the same thing with worldwide IDs which is really why this has been written LS BLK J looks just like LS BLK on the output you can use all the standard command line arguments on it the only bits missing are the file system information because we're not yet logging that but in future we may be doing so so I'll create a device called test one obviously I'm recording the date here because I need that for my parameters later so I've created a very simple device mapper device called test one it just returns IO errors and it's 50 sectors in size that's all it's saying there decided to make it bigger so again using device mapper commands I've increased its size there to 50 thousand sectors resumed it to make it active and put the date on so the thing to note that was between 1242 and 35 seconds today and 1242 and 56 seconds today so the size of this device changed over time so if I run my LS BLK J and I say until 1242 and 30 seconds I haven't created the device yet so it doesn't appear at the bottom of the output if I run it now at 1240 10 seconds later it has arrived and it's 25k in size and if I run it again now it's gone up to just under 25 megs so using that time I'll page backwards you can see the bottom line changes and it's that until parameter that's controlling what I'm seeing there so this is just one application using the logs but it makes it very easy to see so I now have a demo if the video plays we will see if it works on here yeah so this is the old interface for skydive there is a newer interface which displays the devices a different way is that going to play yes is it yes I'm afraid I'm not sure how to hide that bottom bit so this is showing all the devices on the system you can see disks network devices and so forth in a GUI and you can use the GUI to select some of those items and view their properties so up here we've got a mirror device a volume group LVM volume group that's a mirror and on the right hand side there we see a window just appeared with a metadata that's relating to some of those devices so over here we've got an our image device forming part of it mirror LV so if we just click on any of the devices we can see the attributes of that device and if we scroll down there we can see the sector size owner request size and so forth that is active and so on and then underneath we've also been recording some metric information the average queue size read size write size and so forth so we're storing that in their time sequence data as well and then down at the bottom here we've just got a summary of total metrics on that device reads per second and so on just displayed in a slightly different way basically the same data as above so next up here we've in Skydive we have the ability to run queries against the graph database so first of all in the highlights box we want it to highlight any device that meet matches this particular expression and you can see there five devices have just been highlighted after we cut and pasted in an expression that's a query a bit like SQL but it's a graphical query language gremlin and I've highlighted anything where the reads per second is greater than 30 and it's matched those five devices so it's highlighted them if we go to the filter box we can put that same expression in there and instead of highlighting it will remove everything else it will just show the items that match that expression so when we do that we find we're just left with those five devices that have more than 30 reads per second so finally up here there's also on the gremlin console you can write your own queries in here directly and see the results in JSON underneath so that is that same query and that's the JSON data that's come back from the database which Skydive has now displayed and then the final thing if we add one more piece on to the end of that query we scroll back up there here we are of count this dot counts that will tell us how many and it's come back and said five there were five matches there were five devices so I'm going to I'm going to finish there if I flick back to the final slide yep those are some links to the package it's sitting on github there are packages built already for Fedora using the copper facility so you can try them out I'm looking for feedback and to determine whether to take this further or not whether it's useful for people so I'll end it there thank you