 My name is Johannes Meichsner. I work for SUSE and I would like to and I would like to show you how to use relax and recover on your own laptop. If you do not have a laptop prepared already with two virtual machines, you I will show it on my laptop, so you just watch what I'm doing. If you ever think about using relax and recover, it's mandatory, it's not recommended, it's mandatory. To do what I will show you now to you, create two virtual machines on your laptop and then play a little bit around with it. If you play around with it, with your first steps on your server with two terabyte of data, yes, you take very much time, it's not productive. So first, play around with it on virtual machines to get some kind of feeling how this thing works. First, I will tell you about the preconditions for this playing around. How to install, relax and recover, how to configure it and then finally make a backup and recover on the second virtual machine. The preconditions are that you have somewhere an LFS server running in a way that the virtual machines can access it. And to start with it, I recommend to use simple virtual machines. I use it with BIOS. If you like to learn complicated things, try UEFI first, but if you like to see how it works, use BIOS. As I wrote it, a single disk, a usual CD-ROM drive, a single network interface card for your first tests. On the first virtual machine, a small and simple Linux system is running. This is the system where real will be installed and which will be recovered later. The second virtual machine is identical virtual hardware and it's empty. So we will try to simulate to recreate the first virtual machine on bare metal. Of course, it's not bare metal. It's bare virtual empty machine. How to install it? In my first talk in the morning, I already explained that relax and recover, it's the same, consists only of shell scripts. So you do not need an installer or something like this. Just copy the tree of rears shell scripts somewhere. To do it with the newest code, clone it from github into a directory. It gets cloned into a directory called rears and change into this directory or copy it as you like via scp via copy whatever or install an rpm package. There's nothing special. It's essentially the same. You get 200 shell scripts copied somewhere. I already asked, before I started the talk, is someone here who needs to install Ria on his laptop to do what I did. Those who have it, who have their laptops already prepared have now already re-installed. This is true. Otherwise, I would stop here until it's done. Okay, then I continue. Usually you need some other software to run Ria. On Suze, for example, you need a lot of other software parted and all these things, but that's usually there. But what's not usually there, on Suze, could be the LSB release package on Fedora. It could be the package in ISO image. At least for me, it happened on Fedora 25 server, on Fedora 25 workstation. It was there. Un Ubuntu. I was unable to connect to the NFS share, so I need to install some NFS related packages. On Ubuntu, I was unable to create a real recovery system because Suze Linux was missing and ISO Linux was missing. So you will experience this, you run it, it fails, you see, oh, okay, it's just batched scripts. If there is a correctly packaged software package for Ubuntu, it should have appropriate requirements to pull in the required stuff automatically. But perhaps there is someone, but I tested it with the raw installing of the shell scripts, and then I noticed later. Nothing bad, that's how it works. On Suze, I experienced, it's good to have a file etseriaOS.conf. If this file is not there, I do not get an ISO image, no, no, no, no, I do not get an init-rd recreated during recovery. I didn't notice this for a long time because it always works with init-rds in the backup. But to do it correctly and as it should, you need this file to be created. When you install an ARIA-RPM package from Suze, it will be done automatically. On Fedora, I noticed that SELinux is for me, I'm not a Fedora user, I'm from Suze, is a showstopper. I do not know how to deal with it. There is in real some code that seems to work somehow with it, but I want to present here very first steps. To get a result, so I just disabled SELinux, and I noticed that it fails Strangely, and then I inspected the log and I found no space left on the IS messages, okay, and I investigated what it is. Fedora has the usual TMP temporary directory as how it is called. Not on the disk, but in the memory, it's very limited, so it really needs more space than what is there. So you just use another one, export the TMP directory variable to something which is big enough. So now more or less all the prerequisites are fulfilled or should be fulfilled to start and to start to configure, relax and recover. To do this, you start with an appropriate example config file. Now it depends how you installed Rear. I go back. If you installed Rear as an RPM package, you use the files at the normal place in the file system, slash ETC, slash Rear, whatever it is. If you installed from GitHub or by copying the shell scripts into whatever directory, you change into this directory, where now the relax and recovery tree of shell scripts is, and then you do everything relative to this directory. So you change into the directory Rear, and then you leave out the initial slash. So that's what I did on SUSE. This is ETC without slash, so you must decide on your system if it is slash ETC or if it is in the Rear directory, the ETC. So the same with the example configuration files. Assume you are in the Rear directory, you use it with outslash, user share, whatever, an example file, and copy it to the Rear configuration file. The standard Rear configuration file is ETC Rear local.conf. And then you have to adapt it. The mandatory thing that you need to adapt is the backup URL. You set it to an appropriate to your NFS server and your NFS directory. That's this one. In the example files, you find a backup RL but you need to adapt it to the whatever it is, the IP address and whatever it is on your system, the NFS, the exported NFS directory. This is the place where Rear will store its backup and this is the place where Rear Recover will get it. Then I always use two things. I specify a password on the Rear Recovery system. When you recover, you boot a Rear Recovery system. It's not your original system. It's the recovery system and the root password therein I use always Rear. It's not the root password of any other system. It's only the root password of the installation system of Rear. And I'm lazy. I always let the Rear Recovery system get any kind of network interface. You do not need in the recovery system a full your full network configuration. You just need anything that Rear can access this backup, anything what's appropriate to access the backup. So if you have 20 network interfaces configured, one is enough, one that can access this in the recovery system. So I think I'll show it later to you. I think I'll start now with a presentation, how it works on a SUSE system and some specific adaptations I did. Before I mentioned, for Fedora, you need a temporary directory with sufficient space. Because everything in Rear is shell script. All those ETC RearLocal.conf file is sourced as a cache shell script. You can type there any command you like. So to some extent you can do how to call it. Not a static configuration, but a dynamic configuration. But you must try out what works in the Rear Recovery system. A lot of things won't work that work in your original system, but you can. So you can as well type this command export of the temporary directory in the configuration file. It is much better there than to remember such things and type it on the command line. On Ubuntu, at Rear upstream, there is no real maintainer or contributor who tests Rear on Ubuntu and looks that everything is set up correctly. I installed Ubuntu 16.04 or whatever it is, long term support on my system and found out that in the Rear Recovery system something is missing. And I have no network interface. It's not a device file. ETH0 is not there. And so there's a a big hammer as workaround in the configuration file. I just load all modules that have been loaded in the original system. I run a command to to in the original system to list me all modules, all kernel modules, and then I copy and paste that as configuration variable and the Rear Recovery system will load all of them. I think I do not need all of them, but I I don't care. I need one of them. This is Oh, I destroyed it. This is the module for the network interface. And perhaps some, and this one perhaps two, but I do not know. I just load all of them. Then in Ubuntu I found out that UDF demon does not work. Perhaps it's only one line. If you know about Ubuntu, it's perhaps simple to get it working again. Again, I'm lazy. I need some device node files. There's a configuration variable. Before the very first thing that is done during recovery it executes this command. So hard-coded there they are and then it works. This is not there to show how bad Rear is. This is there to show how good it is. Quick and dirty workarounds are possible for the experienced user. If you have been listening to my initial talk you may remember this advantage, this advantage. Rear is direct, straight forward. Very fast and you have direct access and direct control what happens. On the other hand, sometimes you need to be direct. Of course, Rear should at some time be also available in Ubuntu and well tested there so that you do not need these packs. But if you have 200 Ubuntu servers, canonical servers and you do now you do not need to want to bother with how to do it exactly right in Rear. Just do it quick and dirty for you and you get your things done. Now it's time to show how it works. Pressing the right key, going here. Like in these cooking shows, I don't know if you know these cooking shows. I have prepared something already for you and that's it. What I have here, I have some virtual machines. No, no hoover, mouse hoover. I have a Fedora 25 server, I have a Fedora 25 workstation. I have, this is from previously, I have a less 12 and I have a Ubuntu system. So the generic one can be deleted. This is from, yes, yes. So now it's better. And this less 12 system is running here, the big terminal, the big X term. I installed Rear, I think from GitHub or whatever in a directory called Rear as user root. I configured it, I have it somewhere. No, I show it, first I show it how it works and the boring configuration later if you have questions. Then I call the main script, user has been Rear and I personally, I always like full velocity. So I use all these debugging options, minus small d, minus capital d and I let it make a backup. So now you'll see it's doing something and the backup will appear. What it, what Rear does now, first of all Rear creates its recovery system user image. That's the messages of Rear, copying files, copying binaries, copying kernel. It gets copied into a temporary directory somewhere. A lot of files get copied there and from this Rear makes a bootable user image. This will become or this will be the recovery system that you boot later. What? Ah, yes. Okay, so great that it happened. I know it, but so yes. Rear creates, I can use the mouse, that's great, could not copy. Rear creates a temporary directory and it puts all the things and now it tries to copy it to the NFS location. So let's look at the NFS location, we can see how it is. Can you read it? It's a smaller font, it's okay, I think. It's okay, it's not very perfect, but it's okay. So it copies into a directory that is named as a hostname. This one is in blue above the same, ls minus l, linux, so. And because I made this little mistake that in the morning I tested the recovery and it works, but then always QEMO wants to have access, wants to access this file and it always asks me QEMO whatever of this virtual machinery environment to modify the file permissions to access the file and it did and now the linux iso is owned by QEMO. And here Rear runs as root, but the NFS directory is not exported with no root squash. So on the NFS server the root appears as userNobody and nobody can not overwrite QEMO's file. It's nothing to do with Rear, it's my fault, so we do the following. We move it away, so I move it away only not to hack something, so to get it clear from the very beginning, ls minus l. So now we have no directory with this hostname and we can rerun the backup, the same as before now it will work. While this is working, where is it, cannot move. It aborted when it tries to write the iso image of its recovery system. And after the iso image was created as last step Rear and backup makes a backup of the files, then the backup.tile.gizet will be created. So making iso image now to NFS location. Now saving something to NFS location, now the iso image is there and now the backup will be made and you can watch it how it grows. Unfortunate to read. So now the backup will grow there and while the backup is made you can ask questions if something is not clear up to now because we have to wait. Yes, yes. In my first talk I did not talk about it but there is a configuration file where Rear is configured by a lot of more or less global shell variables and the defaults are configured in a file usershare Rear.conf default.conf if you have reinstalled or you do not have a laptop. So in this file all default configuration variable settings are there and over the time it gets better there are explanations what each setting does. And there is a backup I cannot remember names a backup underscore type or whatever it's called and you can set this to incremental or to differential or whatever it's called. So if you have if you do your full backup on sunday and then each day an incremental backup you have next sunday one full backup and six incremental backups and during restore you need all of them if you do a differential yes imagine just imagine the word then you have one full backup on sunday and six differential backups but each one bigger than the other one and you only need the last one. So both is possible and another interesting thing this incremental backup is only possible with or only tested by me this Rear's built-in tar backup method. Perhaps it also works with earthquake or whatever but I never tried it but Rear has support for a lot for really many external real backup solutions for example Barrios but I think Barrios has no the Barrios implementation in Rear is strictly separated from the tar implementation so I cannot tell if incremental backup works with Barrios or if Barrios itself just works different and does not need it also and and fortunately we have Barrios people here who could answer such questions. Okay I must repeat the explanation from the Barrios expert for the for the recording and the Barrios expert explained Rear has two parts first it makes the rescue the recovery system and as a second part it makes a backup you can leave out the second part then the command changes from mcar backup to mcar rescue and Barrios when Rear is used together this Barrios Rear is only used for the mcar rescue part for the recovery system and not for the backup part so with Barrios you do your backup with Barrios and Barrios supports incremental backup and a lot of other things so there is no problem just use a professional backup method that's recommended in professional environments anyway not to do it with tar but to use a professional backup method and let Rear do what Rear is designed to do the recovery system and let the professional backup solution do what the professional backup solution is designed to do. Okay so now we have this one now we have on the NFS server a backup and an easy image so to simulate that now the system has somehow gone I shut it down and I create a new virtual machine this is the equivalent you get a new piece of hardware because the old piece of hardware has totally crashed so I create a new one generic on the NFS server I select this directory so the new virtual machine will boot from the easy image of that contains the Rear recovery system here is the emulator would like to own this file so for the next time I could change the permissions of the file and then this message would not come but that's that's the reason we call it a little bit better less 12 sp2 next Rear so it's only well these are all only steps that I prefer to do to get my virtual machine appropriate for for this create your second virtual machine as it is good for you the only thing which matters is that it boots no that it boots I had okay it has stored it somewhere else it's not a CD-ROM so now the second virtual machine has booted no this is a boot loader of the Rear recovery system on the second virtual machine this is a relax and recovery boot loader screen by default it boots the first local disk there's nothing on the disk it would not help here but to be safe you cannot too easily recover your machine because you remember my first talk recovery means reinstalling from scratch so if you do this on a not on a new machine so but on a on a machine that is in use you get a reset to the backup of course that's what happens below you can see some the command line the kernel command line you can even press the tabulator to edit this command line and append a parameter if you like so or you boot it strange then the unit ID is loaded here you see a progressing dots and dots and dots and when you had your talk there was three dots and nothing happened you you use use a different boot loader yes and it's of course and it's much better to not tell the user you're still living it's not your problem it's the boot loader is just silent and perhaps you can configure grab 2 to be a little bit more than both what it's doing so no this is not exactly right I do not know where rear has it from this is not welcome to Susie Linux enterprise server 12 sp2 no no this is welcome to the rear recovery system I guess it has copied etc issue etc whatever so that you see this message interesting I no no no it's always the same I never noticed it but it's the wrong welcome message welcome to relax and recover for doesn't matter yes of course this happens while rear makes its recovery system it copies a lot of files and in one of these files there's this message it is the motto of the day you find it and on the other hand it's not too bad first of all it tells it's relax and recover and it's relaxed and recover for this system so it's we can leave it as is and lock in as root it has no password again we are not on the on any real system here in the rear recovery system so I always type I have config to see I have an easy and a network connection why don't just a moment I must resize it to fit on the screen so and now let's do the recovery now I do not rear minus small d minus capital d recover so things happen fast yes I don't know it know how fast it goes usually it's fast I am in my first talk I said rear is fast and now I must prove it yes it's really that's that's the important thing that's the only important thing you have 200 servers and your foobar server does no longer work and someone asks you oh my foobar so no the foobar service does no longer work and then you know the foobar server it's responsible for this oh shit the machine is broken I don't know something is really corrupted don't know you have in your rack a lot of separate spare machines just fire one of them up and get it with the backup of rear and you are done yes making the unit rd all the most takes longer than restoring the backup but now you are done system is back again you can reboot now it's terribly hard to reach can anybody see something it's not good but it is as it is so what does rear first of all something then in rear what is called layout so what layout in rear has a special meaning it means the disk layout the layout of the partitions it's called since some years layout so how many partitions do you have how are they connected the block devices the lower level block devices the higher level block devices until you finally get block devices where you put file systems in LVM could be in between or something like this rear supports it so it creates partitions it creates file systems swap and then disk layout created full stop at this point the hard disk is prepared and I must show it to you and all partitions in this case it's simple it's only one is mounted at a mount point in the currently running recovery system and this pound mount point at rear is called MNT local doesn't help very much at this mount point MNT local the backup will be restored so all files from the backup end up in MNT local in the rear recovery system here you have now everything restored because the restore is all has already happened and therein is your new created system do we have find mount yeah we have even find mount in the recovery system that's not yeah you see that you in the recovery system your hard disk partition device node file devsdh2 this extended for file system is mounted at mount local and there there is your new system so we reboot now everything gets unmounted of course that's the main task of rear is to exactly such things rear will reinstall the boot loader it has three big steps the first one is recreate the storage access make partitions make LVM make file system and then mount all together those that you have at one mount point the whole your whole hard disks then uh let's say dump all the files therein and finally reinstall the boot loader in a change route environment of course in this recreated system and this way you get it bootable rear does not explicitly deal with the master boot record so there's no code but it scrap grab install must do the right things to get it bootable again so there is an execution of grab install in this yes yes there's an execution i removed the lock to recreate the unit already and to reinstall the boot loader it's it's it's a it's just a normal installation but it's an installation via bash scripting so there's no there's no difference between there's no no generic difference between rear installing a system and just um um anaconda whatever it's called the only difference is how to how the payload gets there for rear it's restore of a backup other tools install packages whatever else but the the major thing is an installation so i can look in and voila no big surprise no wrong window so the system is back it looks similar like the first one and i get rid of this so now let's try the same with fedora surfer there's no much not much difference this may be a bit of a naive question i guess but is it possible to tweak the source of the recovery before doing the recovery so let's say that you want to do the recovery after a disaster on a different machine which is not exactly the same one so maybe the disk is slightly smaller or you want to install it on the second one is it possible yes it's of course possible in general rear is bash scripting so everything is possible that that you can whatever you can do on the command line is possible the only question is how complicated it is and in my first talk i said uh ideally use the same hardware but if you have a little bit different hardware for example if the disk is bigger rear will automatically detect it and tell you all bigger disks and then you say yes and it will work if you have if you had before let's say one terabyte disk and now you have uh no yes you have two disks this with this 750 gigabytes then you would do a little bit more because rear cannot guess what's the right thing is and there's no code in rear to guess such situations imagine you have had before a network interface card of two type whatever and now you have a totally different one then things can get really complicated perhaps you need a kernel module that is not there in the recovery system i think all kernel modules are there okay gratzchen confirmed all normal kernel modules are there but you have a brand new very special hyper performance thingy which needs firmware then things could become complicated you have a kernel module but you do not have the firmware for it this is not the end of the rope because uh you if you set up rear with uh remote access you may be able to get this firmware from somewhere and put it somewhere and make it work if you are clueless it's a dead end now it doesn't work but you you need a file system setup command you want to switch from extended four to butterfs rear is small also look at my first uh there's a the presentation is online look at my first talk rear doesn't if there is no butterfs file system used then rear does not include the butterfs tools in the recovery system and then things get tricky more complicated because you cannot call mconf s dot butterfs know that file but if you know this in advance you can tell rear to include this or that program or this or that directory into the recovery system and finally i implemented something uh said you can do this more or less automatically as a very first step this did not happen here uh rear if no in the recovery system you have the same rear configuration file it is your local dot conf so you can change it if just a few adaptions for example your for whatever reasons you cannot access the nfs server from the new machine so you can move the backup whether the rear things to a place where it can access it and and change the nfs server the backup underscore url parameter to the new one so you can always do something and um what i implemented is that you can even download you can do it manually but you can it's in rear that rear can do it for you automatically to download a targi set and just unpack unpack this in the root of the recovery system so you can change the recovery system as you like you can of course damage it if you like or you can so this is intended exactly for you use case target hardware is a bit different you need something special you made a backup it's now on the nfs server and they're in there is the iso image and for whatever reason you do not want to recreate it or you cannot recreate or whatever then you can later add something but the crucial point is if you are clueless and do not know about your systems then it gets really hard but if you know you can do so so many things so the more you use it some more i think the more you use it the more you like it when you use it for the first time it may fail this is and this little annoying thing and then you learn ah i must do this and that and then suddenly things get really easy and you have a lot of more possibilities then with some kind of fixed solution you're welcome did it work for you no ah yes yes windows oh now bloody news great newest greatest hottest not yet well tested feature actually bloody me i must repeat all this so if i tell it because of the microphone nobody will hear you if you match if you match such a case you can just please a moment i must repeat your question and his answer for the for the recording so the question was is windows supported the short answer is yes the long answer is yes but um there's a brand new backup method which is not a usual backup with very usual backup i mean backup of files this new backup method implemented by bloody mirror is called block clone and what it does is it it clones a whole partition your windows partition either this dump disk dump or with a more intelligent uh what is called an ntfs clone because rear runs on linux it has no idea what there is in this ntfs thing and ntfs is complicated and you do a bit wrong and it falls apart so it clones the whole partition and it restores the whole partition so windows is supported but only as dual boot because you need a linux system where you run rear and as far as i know it does not work it cannot work when windows is running so you have a dual boot system you boot linux and then you block clone your windows partition and also the master boot record is taken care of and then you as dump data blocks you restore it it must be restored on the exact byte where it has been on the disk so small uh movements is deadly for windows so uh but it should be supported and i appreciate your feedback and your testing uh i it's a question was if uh currently one needs to use a running linux system to backup windows partitions that are not in use the question was if it's possible to use a live system a live linux system where rear runs and then do the same i cannot answer this question right now and i have no idea if there is sufficient user requirement perhaps it is and it's open source and then it can be implemented uh grazien can rear run as as a live system on a live city is this actually it is a live image more or less your boots uh it's a rescue image yes so we are almost there so if you contribute yes it's often so the case if you think a little bit so you notice 99 percent is already there you just need to put some pieces in the right uh in the right connection and then things fall together and it fall apart i think it's uh good for operating system backup yes not for the data backup but it's good for production systems just making up with relax and recover the operating system yes leaving leaving the application data okay i would like to repeat something from uh uh d rlm uh main what is i'll call it from a drlm member uh i mentioned that tar is not a good backup solution or something like this for the plane operating system tar is perfectly okay but if you have terabytes and then of huge application data then perhaps tar is is not the right thing and since read 2.0 there's a new way if you watched it there was one big backup and it was restored imagine you have 10 terabyte of database backup in this tar in this one single tar then and you restore on a powerful machine then you do not utilize the power of the machine because there's only one data source and you and you you make the backup so multiple backups and restores are also supported now that you can have first you reinstall the basic operating system and then you can restore multiple backups at the same time so there is no limit it only the only limit is how fast your hardware can do it so time is up i'm finished thank you very much for attention and please try it out