 Hello, I'm Jörg Steffens from Barrios, from the Barrios project and in my presentation I like to give an overview about Barrios software, Barrios backup solutions for people new to Barrios. Yeah, so what will be covered is... Don't move the laptop, okay, so what will be covered is what Barrios is about, what this is, what this feature set is about the Barrios architecture, the installation of Barrios, the basic workflows and then also get some configuration examples and show how to extend it. Also I will give an outlook to the roadmap for the upcoming features, for the upcoming versions. So, what is Barrios? It's a multi-platform backup solution working over the network. It's 100% open source, obviously, otherwise we wouldn't be here. The source is available at GitHub, the core components are written in C, C++ and it started as a fork from Becula in 2010, maybe some of you knew Becula or not, okay. The first public release has been 2013, this has been Barrios 12.4.3 and from then on we have done a major release every year and the current release is Barrios 16.2.4. Some people ask what's the version someone come from, so the mouse is also visible, yes. So 12.4 or 16 is the year of the code freeze and the second number is the quarter of the code freeze and then you have a minor numbering for the different versions. So the current version has been the code freeze in the middle of last year. Multi-platform, the Barrios project itself provides packages for all major Linux distributions or a lot of Linux distributions, packages for Windows, 32-bit and 64-bit, Mac OS lines and on customer requests Barrios.com also provides packages for FreeBSD or for some UnixX like AIX or HPOX or Solaris. It's also integrated in some distributions. I'm aware of Arc Linux, Debian, FreeBSD, Ganto and because it's included in Debian it's also available at Open2Universe. If you're asked what version you should use, the version integrated in your distribution or the version from Barrios.org, it depends on the features you want to use especially for Debian, Debian Stable includes the Barrios 14.2 version currently which is a working version of course, but some features have been introduced later on so you should decide. If you set up a new server, maybe you start to give it a try with Barrios.org. How does an environment with Barrios looks like? You have your backup servers components somewhere here. On every client you want to backup, you have to install the Barrios clients called Barrios FileDocument because it's backup the files from the systems and you have some special plugins for backing up for example the Ember environments or MS SQL cluster or backup NDP. Also, if you may use some cluster file system like ClusterFS or SF then these normally are highly redundant but nonetheless it makes sense to also backup your cluster file system. Where is the data stored? Most people start storing them to disk on files again. A lot of environments in larger environments use tape, auto changers, tape robots, it still makes sense and has some advantages or you can also store them to some the same cluster file system like ClusterFS or SF. You have your central components and you have your one-time control via a web or text-based console. I will come to this later on. Features, hopefully all the features that you expect from a backup software are included. Barrios are distinguished between different backup types like full backup, differential backup and incremental backups and newly introduced is the always incremental. So in this case on your client, so the Barrios client which might be your server only incremental are one and only the incremental are sent to the server so minimizing the amount of data sent to the server. But if you're using always incremental then the server size has more to do like consolidating your jobs and recreating full backup sets again if you configure it this way. But it's not only about backing up the data, backing up the file but also about managing of your backups. So normally backups are done to volumes which might be a tape, might be a disk file or something like this and you assign special properties to these volumes like the retention periods meaning you define for a volume how long you want to keep the data on this volume and Barrios will care about that these retention periods are fulfilled. You have a flexible scheduling for your jobs and also quite flexible with your network setup. The different interface that I already mentioned you have the B-console text-based interface. We have the Barrios web UI. It's a PHP font and I will show it later on and from the regular fork we still got the but QT-based console which is also quite in some use however we hope we have now integrated all the relevant features from but into the web UI and therefore marked but as deprecated for Barrios. If you're doing your backup of course you want to secure your data but also the backup system itself should be quite secure and here we have a challenge response authentication mechanism for the transport, for the network transport it can be configured to use TLS. Data can be encrypted on the lines in software which means the advantage is it is encrypted on the client and the server only sees the encrypted data. So this is good for security reasons. It might have some disadvantages if your client really crashes and the Huttus is broken because this is the only one who can decrypt the data but you can still define a master key so that you can also decrypt it on some other places. It's your choice how to handle this. Making this on the client in software has also some other disadvantages for example if I'm a full backup job once on my laptops on Friday then of course my room is stored on an encrypted file system so the kernel decrypts all the data then Barrios encrypts them again and moves them over to the server. It's more than a good time to make a lunch break or something like this because you will notice this. On the other hand side you can only also encrypt data on the back end for example if you just store it to this you can use a normal Linux encrypted file system for this but also on tape since a common tape technology is LTO this is mostly used now and I think until LTO 3 or 4 they offer encryption by default and then it's done in hardware so Barrios keeps the management of the keys to encrypt the data and the tape drive itself will encrypt and decrypt the data that you store to it. The advantage is it's there, you have already bought it and it doesn't slow down your backup. Of course it's something else if it's stored encrypted on the back end or on your client but nevertheless. Barrios has quite some flexible logging mechanism and by default you also got an audit log so a special log file which tells who did connect to the backup server and issued what commands on the command line. On customer request we also added a secure erase command this is all files that are created by Barrios will then also be deleted with a secure erase command like Vibe for example which hopefully guarantees that the file is really deleted from your hard disk. Some financial institutes have some requirements in this respect and for us it has been easy to add. We have quite powerful ACL support, I'll give an example later on and if you want to backup all your data on your client and also restore it to the same client. Of course your client needs to have all file permissions to do so normally on Linux the file demand runs as route and has access to all the files but you can also configure it in other ways that it may only be able to do backups or only to do restores or run as a different user and you can specify the command that allowed to be executed by the file demand so you have some security options included. Features you can build your scripts around Barrios there is a special presentation about this later on you can extend the functionalities by plugins you will also need so if I will show some examples later on if you run this environment you can also define quotas for specific jobs so that specific systems only can backup this amount of data and otherwise you have soft quotas and hard quotas and it also integrates with some other open source software for example Loelax and Recover. Barrios architecture we have a picture before but now I want to show it in a bit more detail so in your Barrios backup environment you have one central director who controls all your backups when they are started and things like this you can have one or multiple storage demon and to each storage demon you can attach one or more storage devices like I said before this tape cloud storage something like this and then you have the lines connected to it the director itself stores all the persistent information to a database called catalog and yes you can connect to the director why is it so called Barrios director console interface so the web UI uses the same command sets as the text console why having different storage demons if you can attach different storage areas to the same storage demon good question the question is why you have more than one storage demon because you can attach multiple storage devices to one storage anyway the answer is because maybe you have different locations so you have one location, second location and maybe even if you have different data centers you can backup all the data to one storage system but if then you may get a disaster and building bonds down you can have also your job scheduled to replicate the data to another storage demon so the third demon can be remote there are all this is remote and the communication lines you see here is all TCP ok ok a bit more detail the file demon you want on every client you want to backup it can read files, it can write files, verify files it's not only reaching the contents but also the attributes, ACLs, extended attributes on Windows it's also responsible to do the VSS snapshots, you can configure check some calculation it makes sense if you really want to check if the file content has changed it's responsible also for encryption we noted this before and also for compression and it can run scripts that you defined on the server for the client and be extended with plugins continuous backup is not supported like via iNotify or only in special respect for example in the ember environment we have changed block tracking and then we can do something like this otherwise you have to create a plugin to fulfill this because you don't have a standard mechanism that we can use on all systems but yes we would like to have this feature storage demon I mentioned this before you attach the different storage devices and also it's just like if you have your tape library do the barcode labeling and keeping care of this as said before if you have multiple storage demon you can migrate or copy jobs from one storage demon to the other also changing for example the compression or things like this via you migrate the job and also cares about errors like tape alerts and things like this the database called catalog is an SQL database supported are Postgres SQL, MySQL, SQLite if you don't have specific preferences just choose Postgres the director handles the data in the catalog do all the media volume handling, pool handling, scheduling triggering of the jobs so starting the backup jobs caring about your restore jobs and things like this yeah keeps messages that the two receives and also are able to run job on the server network connectivity the network connection between the different components are not permanent but as they are only established when they are required and normally it works this way if you have a new backup job then the director tells the storage demon that soon there will be a client which connects to the storage demon to storage data there after this it connects to the client and tells him you should now run your backup job and start to this storage demon so the director is not really involved with the data anymore but only for establishing the connection and doing the authentication between the components the client needs to see the storage demon they need to be able to talk one with the other client and storage demon you can't have all the traffic go through the director no, that's nothing that you want to do but what you can do because for example as I told now the director connects to the storage demon connects to the file demon and the file demon connects to the storage demon this doesn't work in some environments for example in your DMZ where your backup systems are stored allocated in your central in your local network and you have a server you want to backup in your DMZ then the DMZ servers can't connect to your local network so for this we have the passive client in this case initiation is the other way around so ok, don't get confused by this the the storage demon pulls data from the file demon so you have to expose the file demon in that way yes sure so the other system has to be able to connect to this and the file demon would be talked there will be a storage demon that will connect to you soon so the other way I want ok the other way also on customer request is that the director can't connect to the file demon in this way you can use a client initiation connection where the connection will be established from the client to the director in this case it will be a permanent connection, network connection and if the director has done a job for the client it will reuse the already existing connection ok installation of barriers as we started this project we want to make sure that installation of your backup system is quite easy and quite fast to handle this so these are the steps to get in working in backup environment in limited sense first you have to decide what database you want to use for your backup system then you add the barriers repository if you want to use the package for its barriers org you install the endowed only the barriers package which is the meta package which pulls all the other packages and then again depending on your distribution you have to run a few scripts to prepare your database for example in divin or buntoot this is done automatically in these scripts after this you can start the daemon you start the director you start the storage daemon the file daemon is already started and then you have working barriers environment which runs a job every day to at least backup itself so you have a working environment and working configuration and you can just extend it you mentioned that you can have encrypted backups and the keys are managed in the barriers database how do you backup encrypted backups I mean how do you backup barriers in encrypted backups or how do you install them so the key management has been if you have tapes and you see a tape SCSI crypto settings for this and in this case then the keys are transferred to storage daemon but if you want to make a recovery because of barriers yes how do you get the keys out of the storage yes for all these suspects for most of these suspects we have command line tool so you mean the disaster recovery where your backup server all the questions for each job for important like backup to your own database there will be an extra message generated normally sent to the administrator which have some information to restore this single job by command line tools from the medium itself which contains the keys I'm not sure about the keys in this environment but you should take care that you have the keys somewhere stored but probably they are there I had to check how to do this probably okay packages we tried to split barriers into functional units you get the packages from downloadbarriers.org barriers release latest as said if you want to have a full environment you can just select install the barriers package it will trigger it will have dependencies to the other packages that it requires normally it also chooses the correct database backend packages if the database is already installed on the same system if not you have to verify that you installed the correct database backend package on the lines you only need to install the barriers file demon and on the server it also makes sense to install the barriers web UI if you want to have the web UI running we built our packages using a private instance of the openbuildservice.org so openSUSE build service this is in a sense you see here some of the distributions most packages are built directly from the main Git's repository Barriers Barriers some other like the documentation are only built on Debian and then exported to the web server and some plugins just for specific platforms for the requirements for plugins what's also quite interesting that we can also use this to build the windows packages we built them by cross compiling them on slas12 you see here we built the packages winbarriers for 32 bit and 64 bit and install and so on so what architectures do you support? okay these are only builds for Intel and yeah of course it's integrated in Debian and we see that it's also used on other architectures more question? well is it backwards compatible for instance if I use Debian you said that it contains an older file and can I use that yes so the question have been can I use the current Barriers director with older version of the file team definitely you can use all Barriers versions against the new the current file team you can also use the backyard lines at least up until 5.2 but then of course you cannot use all the features yeah windows lines I'm not sure if somebody is using windows here it's an NSI installer and then you can select the components that you want to install and yeah run it and you already configure it with this frontend here normally you only install the file daemon so the client components but it also includes the server the storage daemon so you can do one your full installation your full Barriers environment on windows but yeah you shouldn't it also includes the web UI there has some features like that the windows firewall is configured accordingly and has also silent install options so if you not only want to install it on one system but on a bigger set of on more windows systems then you can do this with command client and command line arguments configure it management windows software we often use Opsi the open source solution option to manage software on windows and because we like this we are already creating packages usable with Opsi we have also debug packages which includes source code and so if you run this on windows you can use your gdb on windows and can debug it there which you also don't want is it possible to backup or abandon the restore of windows? no not on windows yeah we have done some experiments with this but in principle you can backup the full system and restore it but before this you have to initialize format the hardest accordingly and things like this so in very simple setups where you know how the hardest did look like beforehand you can do so and we have done some experience with this but as a general solution good as relax and recover no relax and recover for linux yes how many clients can a daemon serve? yeah we have environments with at least thousand clients or more just one daemon server the director just one director but multiple storage daemons the data is sent to and what happens if a daemon the client dies? no if the storage daemon dies then the job will fail normally the director would assign another daemon for the client? no no no not automatically so if some system broke one of your storage daemons then yeah you have to fix it and reconfigure your jobs so the jobs that using the storage daemon will fail during this time do you have a change for the director? no I think we have done this in some projects but yeah not as a common solution ok workflow so if you have your various environments set up you can start the command line tool B console which opens the TCP connection to the director to the authentication there and then gives you an interactive form to type in your commands to interact with the various daemons type in help and you get the list of all available commands if you then for example want to start a job run a backup job you type in one then it will show you the list of drops that are available here we have chose four for backup client form which is the default job if you select it it will give you an overview of what it does intend to do so let's use the backup client one job name it will do an incremental backup the client is named barios fd file set to find the files that should get back up and here the backup will be stored and it's really done and then you have the choice to modify the settings or run the job if you say run the job then a new job is generated and a new job ID and it tells you the job ID for this job and also tells you ok here something happens I have got new messages oops you can then type in messages and see what you want to tell you and in this case yeah what's just important it has been the first job on this client so it says ok you have selected incremental backup and any full backups before so I upgraded this to full and to a full backup it tells you what storage device it will use and what volume the data will be written on and when it's finished this has been a small job it gives you an summary about what has happened how many files have been saved and what amount of data but the important part is backup ok so it did work fine if you start the job for a second time it gets the next job ID same before just it doesn't upgrade to full because you have a backup job before and the summary says ok it's an incremental it's only 10 minutes after the first one so nothing had changed so I don't have written any files and not 0 bytes and therefore I haven't used any volume to store the data too but backup is still ok of course I fulfill the job you can do the same in the web UI so you have typed in one in the B console to get the list of clients where to backup in the web UI you have this job action view there you get the list of your jobs and then here you can start the job on the specific the specific job if you would have typed in list jobs on the B console it would show you something like this it's readable more or less if I wasn't in the way so here you see ok there has been a job ID 1 2 3 3 is the job we started before the backup client 1 the job has been named backup client 1 the client has been backup fd it has been of a type backup it has been a full job and the number of files we have seen before and here again succeeded and then you got the choice to rerun the job which normally makes sense if the job did fail for some reason and you have fixed the reason or to get list of the file in the restore browser of this job the second job incremental with zero bytes and so on it's also available here can you also restore a track file from the B console pardon? so you said you can see the files can you restore any selected file from the B console so the B console has been there first and offers all the commands and the web URL is only frontend on it and uses the same connection issues the same command you type in in the B console list jobs I had before if you go to the job details you can see the job log as we have seen before messages so if you type in in the B console list job log you would see roughly the same content here and then of course important for backup solution is also to restore the things you select something or go to the restore browser you select backup from what line you want to restore then you get a list of all the backup that did run for this client then you can decide that you only want to see this single job or want to merge it up until the next full so only if you have done an incremental not only the file that I include in the incremental but the incremental before and the incremental before they have been the full job you can select where you want to restore it do you want to restore it on the same machine or another machine and what should happen with the file there should they be overwritten by the backup or should they be placed somewhere else or whatever ok this has been a quick overview about the workflow the barriers configuration so with the web URL and the B console you trigger actions but the configuration like adding new jobs or configure file sets you have to done in the configuration files each demon has its own configuration file with the current version we split them into a directory structure with resource types and then specific resources for example the the file set defines what job will be back up I just set down because maybe you see them better ok here we have a file set resource we give it the name linux all which means we want to backup all the files that are included on the linux system therefore we define the root directory to be the source of the backup but the default setting is that it backups only one file system this makes sense because you don't want to backup proc or this or tempfs something like this but here in this case we want to have a configuration that backups all the files so we choose another way we start at the root directory say use all not only limit to one file system but backup all the file system we have butterfs, x4, zfs and yeah backup them the list in the original is much longer but I think you get the idea for this and yeah excludes all the files that are at the temp directory it's a file based backup in this case so it's back up the files that it sees at this situation other way is if you only manage your central backup environment but want to let the administrator of the different lines decide what files to backup you can use the construct like file this is backslash less than and then a file name no it's not less than but it should be but I wasn't able to get it in the presentation so it should be like this look it up in the configuration then on the client machine the administrator of the client machine has the option to specify what files or what directory she wants to get back up and so you can handle this this can be combined easily with this quota so you can limit the amounts of data which administrator can back up so there is no integration with like cfs snapshots or dqfs snapshots or something similar no not in this case it's just file based and you have different plugins for other things we come to the scripts later on so scheduling you can define schedules this is also a default schedule delivered with the default configuration and I did say run a full backup on every first Sunday of the month at this time run a differential backup on the second to fifth Sunday of the month and from Monday to Saturday just do an incremental backup things like this a client will look like this you give it a name, you give it an address normally DNS or IP address until you define a password and then a job is just a combination of all the other resources so you give it a name for the specific job you say what line should be back up what file should be back up when it should be back up to what storage should get back up how the messages that are created are handled and where to store them on different pools there is no configuration for volumes for single tapes but just a pool and this assigns specific settings to a set of volumes can you specify multiple storage areas or just one can you do this one or if it doesn't work this one just one and in the storage definition storage definition is one per storage demo there can be multiple you can access multiple storage areas but you define it here you can overwrite it when you start a job but it doesn't handle this automatically it just does what you have defined here can I start a backup program triggered by the client for instance if a laptop connects to a network to the home network when the laptop starts back up something similar will be in my presentation about barrier scripting later on today yes but not nothing that is really included some python codes, 5 lines I wanted ok I see 50 minutes left so I do it quick you have your full pool so it's where your full backup will be stored and there you define how long they should get be kept so everything you write to your full pool should be at least kept for one year and you just define this file size or just to file similar on the incremental you just say incremental I only want to keep for at least 30 days and so on this has been now about barriers configuration but only if you currently only do this for the server itself if you want to add a client you have to install the barriers file configure it on the client and on the server and then yeah we started on the client so go through this add the barriers repository on the client install the package barriers file daemon then since the current version you can on the server side issue a command configure add a client with this name with this address with this password if you missed some some settings here and it will complain or if you did something wrong it will complain this will create a configuration for the director itself it will also create snippet of the configuration for the client so that later on you can just use scp the configuration for the client to the client and then restart the client okay if you have done so you can verify that the client is working with status client you can initiate the tsp connection to the client you can check what files would get back up if I would run a job to do this with estimate listing yeah and so on after this you can add a job you can take a look for status schedule when the the job for this will be triggered and then run the job but these things you can also do with the web UI one script as we have been asked before so you have your normal job and you can add one or more one script to it and you are quite flexible so you define this one script should run before your backup starts it should be executed on the client it should execute this command in this case dump your postgres database to a file and if this fails the job should fail yes no the setting you can't be done but it's only done on the server and not on the client this has a disadvantage that you dump your database and have an extra file which consumes also some disk space but there are works based around it the most basic is a bpipe plugin so instead of creating a file you make your dump your database so for the reader and then say use the standard output and create a virtual file not on the files but in the backup with this name postgres dump sql or whatever you want to name it when doing so run the compression around it so it shrinks the data and it has also the settings for restore so hopefully you can restore your database with this command you have more sophisticated plugins for example for mysql or maria db which integrates perkona extra backup with this you are able to make incremental backups of your mysql database backups and point and time recovery it's a so called python plugin so it's some lines written in python so you specify use the python plugin look at this pass barrios plugins the the plugin name is barrios fd perkona and has this configuration you have this also for vmware as we have discussed before in this case you are able to do incremental backups with virtual machines in vmware it connects to the psphere server and yeah enables the feature of change block tracking so on incrementals it only changes the blocks that have been changed since the last backup also a python plugin this is block based or is this also fire based this is block based so you just pretend like this is a big file sure but we use the vmware api for this and therefore we can do this incremental stuff on this on this agent class do you need only vmware environment to run this test I'm not sure maybe you asked Stefan because he has written the plugin but I think ok next thing is ndmp is a backup interface provided by the last large storage system like netapp isilon and so on and in this way the storage system creates the backup drops and parios is able to handle them they also do this full and incremental backups and also a single file with store yeah this we will skip this we will skip acl as mentioned before it's also used with the web ui and as an example for the web ui you can create a new user say use the profile web ui admin and in this case you have access to all resources in the default installation but and to most to most commands to all commands except of the list of commands written in there or some other case you can define a console for another user which only gets access to client 1 and client 2 and also a set of the file sets and so on so good map what we plan to do for the next major release is PAM authentication because now currently we handle these things by ourselves but PAM authentication in principle it's working since a longer time but we have to change the handshake between the demons so it's a bit more effort but we have an external contribution with claims that he has fulfilled this and we're looking forward to get this integrated I've been asked how large Barrios environments could be so if you have several thousand clients and a large number of files especially we discovered some bottlenecks in the database but we have been able to fix major ones and this is already done for a specific customer right now and we're on the way to integrate it to Barrios master the partner interface we'll get to Core, NDMP I'm not sure if you use this but we are quite busy to extend this because currently backup data is sent from your storage to the Barrios storage demon and he will care about the data but NDMP also offers a mode where it can write data directly to a directly attached tape library and yeah we are quite confident and have also done a prototype how to solve this because this is then much faster because you don't need to transfer the data through the network and like access restore just fastens single file restore now, have you time for questions? yes, so question the replication for the storage demons is the client sending you multiple demons no, of course you can do it in a way that the client do the backup twice and send it to different storage demons but the normal case is you have a set of rules about what drops already exist and what drops should be copied or migrated to another storage demon so a job wise and you have a set for doing so for scripting how is the compatibility with Bakula? the basics it should work and of course if you extended commands then it will not work in Bakula but the other way around I don't see an issue yes sure, so we have talked from Bakula 5.2 and everything which worked there should still work here can you please repeat the question yourself are Bakula scripts compatible with Barrios? yes do you always need the port open on the VM that you want to backup? sorry, can you repeat this a bit louder? do you always need the port open on the VM that you need to backup? no, you connect to the vSphere server and you will send the data that you need to know so otherwise you have not a port open with your used client you need a connection then it's the other way around so the client connects to the director ok time's up but I'm still here so maybe you can