 Rhaid oedd yn gwneud o'r peth i'r cyhoeddwyr i ddarparu'r cyhoeddwyr, felly ddim o'r ddweud o'r cyhoeddwyr? Er y dylai'r cyhoeddwyr yw'r cyhoeddwyr yn ei ddaf, ac â'r cyhoeddwyr yn ei ddaf, ddim o'r cyhoeddwyr? Rhaid o'r cyhoeddwyr, mae'r cyhoeddwyr yn ei ddaf. OK. What is device maffa? The other point I'll make is I've written quite a lot on the side in case you can't all hear every word I say. OK. So device maffa is a replacement for the logical volume manager that was in 2.4 and 2.2. Who uses logical volume management? LVM. Right, so I need to have some convincing to do as well. Who uses fedora? OK, different people. Evian. Yeah, Gen 2. Yeah, what phone is this? Cycler. Yeah. Hurt. OK. So it's all about volume management. The task here is to provide the generic component for volume management in the kernel to keep it simple, to keep the interface as simple, and to push the complexity out to user space where it's a lot easier to write code. So, as you can see on the third point, yep, is it working. So this was written in 2001 and it's been in the kernel now, must be a year and a half or so. What it doesn't do is it doesn't know about the metadata. It doesn't know what volume groups are, groups of logical, OK, I should start it. Who knows what logical volume is? Right, volume group. Yeah, so a logical volume is a... If you've got... Why are you going to use volume management? OK. Previously, you would use fdisk partition your physical devices. So we start off with a concept of a physical volume and that corresponds to a disk. If you were using fdisk, you would divide it into partitions and the partitions correspond to logical volumes. So when I say logical volume, it's like a partition, but it's a partition of not one disk. It's a partition spread across a group of disks. We take the physical volumes, we join them together into a volume group and then we allocate partitions out of that total amount of space. But we no longer care which disk is being used for any particular piece of data. So when I say what device wrapper is and what device wrapper is not, what device wrapper in the kernel is doing is the logical volume. It's implementing a logical volume just as the kernel previously implemented partitions. But that's all it's doing. The more complicated business of joining the physical volumes together into volume group, how to allocate space for new volumes, that's all done in user space. The kernel doesn't care about that. So this is a diagram showing how logical volume manager is down here in user space. And as with all good open source ideas, you've got a choice. You can use logical volume manager 2, which is what I work on, or you can use EVMS, which is what I get. Device wrapper is the kernel components that implement a logical volume. Both of these talk to the kernel and do their work in user space. On the LVM2 side, we have a user space library here, and this library hides the protocol between user space and kernel and wraps it in some easier to use library calls. DM Setup is a program that gives you complete access to the protocol between these two layers here. So everything you can do with device wrapper, you can do using a DM Setup command on the command line. But it just gives you the basics. LVM2 builds a lot around it, the concept of volume groups and so on, and similarly EVMS does. So what have we got from the kernel? This is a diagram of device wrapper showing all the different pieces of the thing that we've got today joined together, and I've missed a few pieces off as well. First of all, user space. That's what in previous diagrams show. I've put that all in a little log here. The interface we use is IOCTL. Originally, there was a lot of discussion, should we use a file system interface to pass the control commands to device wrapper? We started work on it, so we have both an IOCTL interface or a file system interface. You can choose which to use. We found very quickly that it was 10 ounce faster to write code for the IOCTL interface than it was for the file system interface, because of the constraints of reliability. So we very quickly abandoned the file system interface. The code is still there, but the reason I left it in this diagram is because the architecture has separated the interface from the core with another layer. So we've got an extra layer in between here, which does the common things that are needed for both the IOCTL interface or the file system interface. Now if we decided we didn't need this piece, then these would have been the same, merged together, and so slowly this boundary has disappeared now, as we've abandoned this one. Separately there is a block interface, and this is used when the IOCTL interface is going through. So the device wrapper core, which I'll describe in more detail later. And then we have what's called a map table target interface. If we'd written it today and we'd have called them map interface, when it was written they were called device wrapper targets. So all the code refers to target, which is rather a puzzling name. It doesn't really describe what it's doing at all. Mapping is what it's doing. It's mapping data from one device to another device according to a set of rules. And here are some of the rules that you could use. A linear mapping, which just has to take these blocks from one device and map them linearly to these blocks on this device. Then with mirroring, obviously, cloning the data between the two. Snapshot and multi-path. How many people here are interested in multi-path? Oh, fair enough. I'll say a bit more about that later. Multi-path has four modules at the moment. And these are going to grow in number over the next year. This is where all the activity is at the moment. K-copody is a kernel service for copying data from one device to another. So mirror needs to use this. It's a separate kernel thread. Mirror says, I need to copy this data from here to here. And asks K-copody to do it. When K-copody is finished, it calls back to mirror and says, I finished. And then mirror might decide to ask for support if you copy it. OK. So map devices abbreviate into ND. Not to be confused with the other meaning of ND in the current software rate. A map device is specified by a table saying how to map a range of logical sectors of the device using one of the map links. This will become clearer later on. Submit and copy that. We use a B-tree so that we can have very complicated map links if necessary. So if you wanted to, you could replace all the spam sectors on your device with sectors on another device. And just less individually the sectors that you wanted to replace. It's also been useful for people with Windows interoperability. In that Windows is capable of writing partitions that start at an odd number of sectors into the device. And the existing kernel code sort of assumes that things always start at even sensible positions. But Windows do that. This doesn't care. This will let you have odd numbers of sectors. So this is what I'm saying about a table. You say where it starts, how long it is, how you want to map it, linear, strut and whatever. And then it details the needs of your map link. Throughout the device map, we give sizes in sectors. We decided this earlier on at sectors almost 512 bytes when we refer to a sector. So an example table here would say nought with a length of 3268 sectors. So that's 16 megabytes and then some mapping. And then the next line says for the next 16 megabytes of the device, I want some other mapping here. Will we refer to devices in the tables because if we have a linear mapping we need to say what it maps onto? We have a choice. We could give it's name, their age to be a, or we could give it's major and minor number. The advantage of the major and minor is it saves us having to look up in the kernel what that major and minor corresponds to. And giving it a path name look up could involve potentially going through several file system layers and potentially allocating memory, reading things off disk even bits. It's a lot more efficient to use this. These are some of the map links that are available so far and the next few slides will take you through what you can do with them. The error mapping, a very simple one, it returns an error when an EIO is passed to it. So this would allow you to replace a disk and make it appear reusable to your application. Not very useful. But if you've regenially lost a disk that's part of a larger logical volume or part of some software range you might, maybe it's that you've got ten disks in this volume and you've just lost one of them. There might be something to begin with by still putting together the pieces from the other nine disks and replacing the missing one with errors so you can then read them directly. Or it may be more useful to replace them with zeros. So you can replace an arbitrary piece of disk with zeros. For example, I can create a 16 terabyte device by using that table. Even though I don't have real 16 terabytes, I can pretend I've got 16 terabytes and zeros. And we'll use this one later on. So oligio table here, start this length, write, use HDA and start from sector 384. So when you write a sector 0 of this virtual device, this logical volume, that will translate to sector 384 on the error of HDA. A stripe mapping. So what we need here is a list of stripes. So if this is two-way stripe, two stripes, put a two here, list the two devices, starting from sector more to each of them. And this is saying how wide the stripes are to be. In this case, 256K. Always expressed in centres. So you have to shout, right, this is a crypt, one. So it's a crypt, the data that goes through it. It uses the kernel crypt to any guy, so it doesn't have any encryption itself. It plays on the existing encryption and the kernel. You have to pass it all these parameters here. But there's a pass tool to handle that for you called crypt setup. Of course, you can see what the ciphers are from the proper crypto. It's a block cipher. And you can vary the block cipher by adding an initial vector. So this means you're not going to be using the same key for every block on your disk because that's not going to be good for the cryptographic setup. So the default is what we call play. And it just adds a number of the sector value to the initial vector. But there's been an enhancement recently called ESSIV, which is a more complicated way of generating the IV and avoids watermark problems with this because sector one, sector two, sector three. You just know the relationship between the initial vectors. It's better to jump on one of them first. There's an option for chain mode, but the only one available is CBC. The other one doesn't use an initial vector, so it's just not secure. And you supply the key in X. So that's the example table for this one. Snapshot, okay, we'll get onto some pictures now. Snapshot samples the state of the device at a particular instance. They're particularly useful for taking backups. If you try making a backup using tar, or by just reading through the file system and copying it to your backup media, you're going to find that the data is changing during the backup. And in the worst case, tar can be working down this bit of the file system recording to the backup, carry on down the next bit of the tree. And while that's happening, you might decide to move a directory from here when tar hasn't got it yet, to somewhere where tar has already finished backing up. So about the time tar gets to the end, it never backs up your important directory because you move it out of tar's way. If you do this, it takes a snapshot at an instance. And then all the changes made after that are recorded so that it can recreate what the original file system looked like at that instance. So you have to set aside some space to record what happens when there are changes. It doesn't make a complete copy. It just needs enough space to hold what's changing. So here is how we handle the snapshots. First of all, you've got a logical volume and you're doing some reads and types through it onto the physical volumes down here. OK. And then we've created how. Copy on write. So this is going... Whenever something is written here, it's going to make a copy of the original data here before it lets us proceed. So what we have to do is put some layers ahead of the original devices. So the same I.R. is coming in here, but instead of going straight to logical volume 1, we have to intercept it, and maybe we have to do something with it. Similarly, on this side, we have to track any writes to the snapshots. So we'll deal with reading, first of all, very simple. We want to read something from here. Nothing's changed yet, so we can just read it straight up from the original basis through the logical volume. If we want to read from the snapshots, it's the same. We can just read the data straight through the logical volume and read them. And at this point, both the snapshots and the original device are acting on. Now, however, we write a block to their origin. So the origin says, has anything been written to this before? No. I'd better make a copy of it first. So it tells a snapshot that something is about to be written to this device and exactly where. The snapshot then reads the data from the logical volume and writes a copy of it into the copy of write area when it's finished, tells the origin it's finished, and then the write can proceed. And, of course, if there was a subsequent write to the same piece of data, to the same area there on this, it can go straight through this time because it's already made a copy. So that's the diagram pulling together the previous slide showing all the algorithms in one place. Now we're going to write a block to the snapshot. Well, we just write it to the copy of write area. We make a note of it. We don't have to look at the logical volume here at all. If we've already written to this place, well, we're just overwriting the data that I've commented to. No, that's right. If we're just overwriting the data that's already there and if it hasn't, we just make a note if there was a write here to the same place, it wouldn't have to copy it. When we read the snapshot, we read it straight from the copy of write area, where it's been changed, and that's a summary showing how the IO flows around the snapshot. If there's time, I'll show you some of these directly later. If you have a second snapshot, it's completely independent. You can't hear it right. So the second snapshot is completely independent. It makes a second copy of the changes. So if you have ten snapshots, you have ten copies. So it just doesn't scale. And there is some work going on at the moment by Daniel Phillips to produce a version of this where all the snapshots share the same copy of write area. Next one, Mira takes a copy of the data. We divide it into regions, fairly large, 512k, and we do all our accounting in terms of those regions. So we build up a little bit of that, saying one bit for every region, saying whether the regions are identical or out of safety. That bitmap could be at memory or it could be on disk. And this is, again, it's modular, so you can choose which one to use. If it's a temporary Mira, you'll use the core. If it's permanent, you'll use disk. Parameter lines are starting to get a bit more complicated now. And with multi-power to get a ten turn towards. But it's still nicer being an ASCII. So the disk log needs to know where am I writing this log, how big is the region, and when I start up, is the device already in sync or not? If it's already in sync, it just assumes the data is already identical on both devices. If it's not already in sync, it makes a copy in the background, using K copy P, of all the data from the first device over to the second device. PV move, you use as a mirror. PV move lets you move all the data off a physical volume. So if you want to remove a disk, and there is a logical volume on that disk, you first run PV move to tell it to move the data somewhere else in the volume. So it breaks the data up into sections and then uses the mirror to move the data somewhere else. So originally, your logical volume is on disk 1. Afterwards, your logical volume is on disk 2. The way you do it, you put a mirror in between, you insert a mirror. So the data is now going to here and here, and K copy D is copying the data from here to here at the background. When it's finished, you take it away, and you're back on PV2. Okay, a little bit of multi-path. I don't know whether the writing can be read right in the back, so I'll try and explain. The top half is kernel space, and the bottom half is user space. Oh, I should tell you. Does anybody need an explanation of what multi-path is? Yeah? Okay. Sometimes in high availability environments, you want to double up everything. You want two of everything. So if one of them fails, you've got another one that can carry on working. What multi-path does is give you multiple paths to the same disk. So you might put two separate discards in your machine, two separate cables from there to the disks, and typically you're using five-channel switches, five-channel cards. It's usually quite expensive and quick, but these days it comes cheaper. You can use iSCSI, so you can have your disk on one machine and export SCSI commands over IP to another machine. So your disk will appear to the local. Okay, so device number is this block here. Encodes, of course. Your low-level disk drivers will appear. When your device appears, so a new disk is plugged in, or a new cable is plugged in, or something that wasn't working has been fixed, we use SysFS and the hot plug mechanism to tell you the space that a new device has appeared. And a diva here called multi-path B scans the devices, looks at a configuration file, and decides how to set up a device number to know about multiple paths. So the IO will be sent down one path, multi-path D, device number will decide which path to send it down. If it's an error, it gets an error, then it will retry it down a different path, and it will carry on until it's no path left. I've got a demonstration of this later. SPARS device. This is where we can use our N16 terabyte zero device. I can now create a 16 terabyte file system, even if I've only got, say, a few tens of megabytes. I can take a snapshot of a zero device, and then write to the snapshot. So this gives me a SPARS device. It only writes the data that I write to it. So as long as I don't write more than 10 megabytes to it, the location of those 10 megabytes can be anywhere within this 16 terabyte address space. Or, of course, on 64-bit, I can go a lot bigger. Right. A bit more about the IoT-TL interface. I'll probably go through a bit of it faster now, because it's a lot more for the programs, and then get onto some examples towards the end. So it communicates with an IoT-TL through a MISC control device. Normal major numbers has minus 63, but the 63 is dynamic, so you have to create a device that can control your cell originally. January of this year, we finally added some code, added some logic to Lib DevMapper, so that it would create this device automatically if it was there, and if it was there, it would have a raw miner number that would fix it. So the hacks that the likes of Fedora have to do to create the DevMapper control aren't necessary anymore. It's restricted to Root. The other Root can create logical volumes of loadings. As Alan was saying earlier, this means we don't have to be as careful of our security as we would have to be otherwise. There are some things we do which, if not Root users can use the IoT-TL, then we have to change what's extra restrictions. The actual payload of information is between user space and the kernel of the request. It's a variable size because these tables can be huge. We don't know in advance how big it's going to be. If user space is asking the kernel what is the table at the moment? What is my current set of map links? Then it has to create enough space in advance for the kernel to fill in, if there isn't enough space, then you get this flag, the above fortune set. And user space should then reach out with a larger number. We have to put a lot of care into the versioning. We, at the time we were originally writing this, lots of people have problems with the previous version of LVM. In the previous version of LVM, the version number of the user space code had got to match the version number of the kernel. Because the interface was often changed, whenever the interface was changed, it wouldn't work. This was a problem. If you were upgrading your kernel, and you wanted to switch between two kernels, sometimes you wanted to boot with one kernel or 100 times with another, you had to keep changing your user space utilities, according to what kernel was that. So, instead of that, we put a much more flexible versioning system in, and the LVM number in user space does all the work to make sure the right version of the IACT alcohols is used. So, the first amount available is the end version, and this just tells you the version number. And we guarantee that this will continue to work, even if we change an age number, if we make significant changes to the interface in the future, all the version number it looks at is the first little bit of the request structure. So, if we extend it in the future, that goes at the end, and all you have to do is change the major number, and it will look at this and say, oh, the major numbers are different, I can't do anything. And then it's up to LVM DevMap to say, oh well, the major number is something different. If I know how to use that different major number, I'll try that. List versions shows you the targets available, whether you've got linear, striped, multi-path, it tells you what is currently loaded in the curve, together with version numbers again. If you want to create a new device, you have to give it a name, which can be up to 127 characters, or a new ID, which can be up to 128 characters. For some reason, there's a difference of one here. Something to do with a trailer in null and arguably a typo, but it's two ways to fix it. You can fix the major and minor number, if you wish, but you don't have to. It will allocate one dynamically, but in some cases, like if you've got servers that give out the major number and minor number other in the handle, then you may want, and you're using failover, you may want to have the same minor numbers on both systems that you've failed over to, and they can be read out. You've three ways of referencing devices. You've got the name, the user ID, or the major and minor number, and you can choose. Device number is meant for the user, so that should be something nice and readable. For example, the volume group and the logical volume name. The user ID is meant for the system, meant to be unique, and once you create a new device, there is no way of changing it. So you can guarantee that if an application uses a new user ID, it will get back the same device. Device number is also... Of course, if you can create it, you can remove it, and there's some nasophilus still in here. For testing, we have a remove all option that deletes everything. You can rename, but you can't change the user ID. Obviously, we get a list of the devices, and then you've got to be able to learn the map names. So we now have two slots for map names, an active and an inactive one. When you load the table up, it goes into an inactive slot first. And if there's already one in that slot, it gets replaced. You can, of course, clear what's in that inactive slot. And then the main mechanism for changing the mapping of a device is suspend and resume. If you suspend the device, it will queue all the IO being sent to it. So if you suspend the device map of the device, the IO stops at the device map of the lab, and it waits while you're making all these changes. If there's any IO still on its way to the disk, waiting to be completed, waiting to come back from the disk, then it waits until that's all finished. So there's no IO happening below that lab. If you have a file system involved, such as XFS, it tells the file system to freeze. And that means the file system will stop processing operations. The file system will make sure it's journal is up to date and everything is clean worthy to FSTK it. So when you take a snapshot, in principle, if all this has worked, your snapshot will not need an FSTK, but it will mount to Queen. And the final thing that a resume will be is move the inactive table into the active spot. So it will, by resuming a device, it makes the device active with a new table. Yes, Paul? It's got an enterprise feature, but is this a better way of doing guaranteeing that from the disk? Of doing that, Tom? You should freeze before doing that, yes. And there are plenty of other examples where this is useful. If you're temporarily wanting to suspend something, but you can do that by calling the freeze directly. You have to have to do it through the device. There are other uses of device on things like if you're distributing an operating system to a number of machines in the cluster, or desktops, and you want the wall to have the same file system image of it, read-only file system for your clients. You can use a snapshot type mechanism for updating and then switching between the old file system and the new file system. It gets queued in memory, yes. If there's not enough memory, you start having problems. So firstly, if you're doing this, you want to do it for a very short time. Unless you've got lots of memory, you want to suspend and resume for the shortest possible time. Well, you have to try it. Yes. It uses memory and the release of LVM2 was delayed for about two months fixing this particular problem for PVMove. Because PVMove wouldn't have the system if you had done memory. I don't believe it does today. So we used to have an info call which gives you the basic information about the device. Is it active? Has it got to the table? Is it redone? But we felt that this was being called so many times a bit like STAT. We decided instead to build it into the other commands. So when the other commands return, they fill these fields in instead. So you don't have to call this, especially. The only caveat to that is when VD be the only count we return how many times the device is referenced. For example, if it's mounted that counts as one. But it turns out to do that you have to take a lock and this was effective in performance so we made this option on the mail. But did it back when we were compatible with the new software to say I don't need this information. So we have the status command and this is aimed at the target. It means a mirror can return information saying I'm 50% of the way through my copy. I'm 90% full and you can monitor that and suggest that you need to make a bit more space available. It's a very, very primitive signaling mechanism to tell the user space that something important doesn't say what but it means user space doesn't have to keep polling the curve to say are you ready yet? Are you ready yet? It can just sleep and when an event happens it will be woken up and then it can go and check but this was just something to start with and replace it with something more complicated like that. This one as far as I know is nothing to use it but it's seen a good idea at the time. It gives you a list of all the devices referenced by another device. So ignoring all the mappings in there mirror snapshot regardless of the complexity gives you a simple list saying this user's HDA, HDB HDC, it just tells gives you a list of the devices used without any more information. And then we have recently added a message that basically passes an ASCII string to one of the targets. So we need this in multi-path. After a path has failed and it comes back again user space has to tell the multi-path target and the path is available again. So it sends it to a message like this and I'll show you that in a few minutes. Lived back to what I mentioned earlier the tools I mentioned earlier will be able to EVMS DM-setup is this one? So you just have commands like DM-setup version DM-setup create and one of the more recent ones DM-setup info-c which gives you a column base output. Logical volume management I'm going to skip through these slides a bit now because we're rather short of time. This was a separate sort of one hour talk just then slipped in why you might want to use logical volume management but you can read this slide all the way from afterwards why we went into LVM2 some of the recent LVM2 extensions PVMove is implemented in a different way from LVM1 it works because it tells the device map you need to copy this data from here to there. The device map gets on and does it and it uses this event mechanism for polling so you have a choice if you use something like base-r5 that means ask the kernel every 5 seconds have you finished yet? and the kernel might say yes or no and tell you how far it's gone so that gives you a display of how far it's not going to copy every 5 seconds so of course you use different numbers instead of 5 if you leave that out or I think it's I-naught or something instead it will just poll it asks the kernel once when you're finished if the machine dies during the copy the move is being done by this mirror target with an in-call lock so the information about what's been copied and what hasn't is all in memory but it breaks it down into pieces and once the first piece is finished user space PVMove waits up and writes to this ok that piece is finished tells the kernel to get on with the next piece so this means that you can lose power on the machine reboot and the PVMove will carry on approximately from where it was before we'll just go back to what that's checked and set the automatic game so you have to run a vg if you run a vg change a y part of activating the volume group is to continue with the PVMove alternatively if you do it by hand you could just say PVMove.dev.spd without any more parameters if that's what it was and it would spot oh there's already a PVMove.spd obviously that's the one you want me to carry on running and then this one will create a new type argument gives you access at the moment to the zero and the error so you can create a segment a lot of zeros in logical volume using this and in the future this will be expanded to some other types for example positive mirrors and positive snap shorts will appear as different type arguments here those will be announced in LVM2 I'm going to go right across those what the new metadata looks like where I'll be able to off-skip that some links I think that's like one that's in half a place no, it's right I finished on that so the main list Christof Sallach is on the acrypto so the main list the main list for a acrypto target, the Acrypt isn't that address there are links to everything from here that's the main website for the device number yeah, the acrypt Christof Baraki is the he has got the site for both in half the slides will go this URL and be linked to there after the event is over and I've got back home and then there's the obligatory output for a conference in Swansea that I organised looking for speakers we found a few days in south Wales right, so I'll now demonstrate to some of this so what I've got here if the network is behaving is that what we've got networking here so it should be working so it's just we'll get that that's better right, so what I've got here I'll run through some of these quickly if there's anything I've set this one up earlier just to show you how you can stack devices together that's the list of devices at the moment I've got a logical volume naught logical volume naught major and minor number like that I've got a couple of crypts devices and some ones that have a link if I use here is set up info.sc some of these are all on main so they've wrapped around but you can see it gives you the major number and minor number, the status and you can look up on the map page what these symbols mean there's a live table there the W means it's right the open account each of them has got one thing using it at the moment reference count they're very similar targets just one and the events the events are all numbered so you can avoid racing missions if you want now let's look at the tables oh I should say if you run into you get a more discreet version it just goes up the top of the screen Alistair Alistair could you show us what's in slash that, slash macro oh yes, denim macro so these are the lot devices that's a good idea and you can see the control device control there the idea is set up at the table this shows me what the map is actually are right not very well laid out but you can see that the first one is a logical volume pretty big I think that's about two terabytes so something strained across two devices the major and minor numbers are here so minor numbers are two and three and now we're off the top of the screen but two and three are actually the cryptards down here so that is strained across some encrypted devices they're encrypted with advanced encryption standard and over devices two by three minor number one and a minor number naught and that's the key that's used for encryption we don't hide it if you route you can put it in if you can take it out I think it's quite right that it shouldn't be visible like that because it reminds people that the key is present in the curve all the time that you are using that encrypted device so it is easily obtained so the only way of getting rid of the key is to disable the device on the mount of the file system so then we've got devices underneath so what I've got is a multi-path with crypt on top with stripes on top of that with a logical volume and a file system on top of that so I've stepped up quite a few words I didn't put a snapshot in but I thought that would be enough at the multi-path you can see the arguments are much more complicated and if you're not a computer programme it's if you don't want to start reading all those you won't have this way there's a status amount 3, 4, 5 status it's been finished and this one again is pretty difficult to work out which always are a game interactive that tells me all we have to act so instead there's a tool there's a tool that does it for you ignore the error messages so this tells me more clearly that I've got four multi-path devices there it sends the IO around Robin and it's using these four first if those four paths were all to fail then it would start using the second route instead it would switch to that route or to that route if I put the IO step down again you can see that there's stuff happening on the various devices if I kill multi-path B so the paths won't be automatically enabled I can do dc set up and find it I can use the message command to tell multi-path to fail device ok so if I now run multi-path again well that hasn't worked that hasn't worked it was supposed to show some of the devices to fail I bet multi-path B has just started to get and it's automatically re-enabled but it's still there but if we have to import the cable out ok any questions the next speaker wants to cover you set up yes definitely well any one volume you can use at the same time of course but you can split a volume in half and use one key for the first half of the different key for the second half if you wanted to change so could you change the key of the volume basically the same and yes you could change the key of the volume by and the way to do it is like being moved slowly move the set up and mirror between the new devices take one section of the file first getting mirror between the device with one key and the device with a new key when it's finished mirror you can use instead of the other one and you can use yes you can start with a volume not a crypt in a different way I think someone has written a tool to do that I think it's always from the crypt's web page for that ok and now it's cut