Bacula is a set of computer programs that permits the system administrator to manage backup, recovery, and verification of computer data across a network of computers of different kinds. Bacula can also run entirely upon a single computer and can backup to various types of media, including tape and disk.
In technical terms, it is a network Client/Server based backup program. Bacula is relatively easy to use and efficient, while offering many advanced storage management features that make it easy to find and recover lost or damaged files. Due to its modular design, Bacula is scalable from small single computer systems to systems consisting of hundreds of computers located over a large network.
If you are currently using a program such as tar, dump, or bru to backup your computer data, and you would like a network solution, more flexibility, or catalog services, Bacula will most likely provide the additional features you want. However, if you are new to Unix systems or do not have offsetting experience with a sophisticated backup package, the Bacula project does not recommend using Bacula as it is much more difficult to setup and use than tar or dump.
If you want Bacula to behave like the above mentioned simple programs and write over any tape that you put in the drive, then you will find working with Bacula difficult. Bacula is designed to protect your data following the rules you specify, and this means reusing a tape only as the last resort. It is possible to _left_quote_force_right_quote_ Bacula to write over any tape in the drive, but it is easier and more efficient to use a simpler program for that kind of operation.
If you would like a backup program that can write to multiple volumes (i.e. is not limited by your tape drive capacity), Bacula can most likely fill your needs. In addition, quite a number of Bacula users report that Bacula is simpler to setup and use than other equivalent programs.
If you are currently using a sophisticated commercial package such as Legato Networker. ARCserveIT, Arkeia, or PerfectBackup+, you may be interested in Bacula, which provides many of the same features and is free software available under the Affero GPL Version 3 software license.
Bacula is made up of the following five major components or services: Director, Console, File, Storage, and Monitor services.
(thanks to Aristedes Maniatis for this graphic and the one below)
The Bacula Console service is the program that allows the administrator or user to communicate with the Bacula Director Currently, the Bacula Console is available in three versions: text-based console interface, QT-based interface, and a wxWidgets graphical interface. The first and simplest is to run the Console program in a shell window (i.e. TTY interface). Most system administrators will find this completely adequate. The second version is a GNOME GUI interface that is far from complete, but quite functional as it has most the capabilities of the shell Console. The third version is a wxWidgets GUI with an interactive file restore. It also has most of the capabilities of the shell console, allows command completion with tabulation, and gives you instant help about the command you are typing. For more details see the Bacula Console Design Document.
The three SQL databases currently supported (MySQL, PostgreSQL) quite a number of features, including rapid indexing, arbitrary queries, and security. Although the Bacula project plans to support other major SQL databases, the current Bacula implementation interfaces only to MySQL, PostgreSQL. For the technical and porting details see the Catalog Services Design in the Bacula Enterprise Developer's manual.
The packages for MySQL and PostgreSQL are available for several operating systems. Alternatively, installing from the source is quite easy, see the Installing and Configuring MySQL chapter of this document for the details. For more information on MySQL, please see: www.mysql.com. Or see the Installing and Configuring PostgreSQL chapter of this document for the details. For more information on PostgreSQL, please see: www.postgresql.org.
To perform a successful save or restore, the following four daemons must be configured and running: the Director daemon, the File daemon, the Storage daemon, and the Catalog service (MySQL or PostgreSQL).
In order for Bacula to understand your system, what clients you want backed up and how, you must create a number of configuration files containing resources (or objects). The following presents an overall picture of this:
Bacula is in a state of evolution, and as a consequence, this manual will not always agree with the code. If an item in this manual is preceded by an asterisk (*), it indicates that the particular feature is not implemented. If it is preceded by a plus sign (+), it indicates that the feature may be partially implemented.
If you are reading this manual as supplied in a released version of the software, the above paragraph holds true. If you are reading the online version of the manual, www.bacula.org, please bear in mind that this version describes the current version in development (in the git repository) that may contain features not in the released version. Just the same, it generally lags behind the code a bit.
To get Bacula up and running quickly, the author recommends that you first scan the Terminology section below, then quickly review the next chapter entitled The Current State of Bacula, then the Getting Started with Bacula, which will give you a quick overview of getting Bacula running. After which, you should proceed to the chapter on Installing Bacula, then How to Configure Bacula, and finally the chapter on Running Bacula.
The catalog feature is one part of Bacula that distinguishes it from simple backup and archive programs such as dump and tar.
Verify can also be used to check that the most recent Job data written to a Volume agrees with what is stored in the Catalog (i.e. it compares the file attributes), *or it can check the Volume contents against the original files on disk.
The File Retention Period determines the time that File records are kept in the catalog database. This period is important for two reasons: the first is that as long as File records remain in the database, you can _left_quote_browse_right_quote_ the database with a console program and restore any individual file. Once the File records are removed or pruned from the database, the individual files of a backup job can no longer be _left_quote_browsed_right_quote_. The second reason for carefully choosing the File Retention Period is because the volume of the database File records use the most storage space in the database. As a consequence, you must ensure that regular _left_quote_pruning_right_quote_ of the database file records is done to keep your database from growing too large. (See the Console prune command for more details on this subject).
The Job Retention Period is the length of time that Job records will be kept in the database. Note, all the File records are tied to the Job that saved those files. The File records can be purged leaving the Job records. In this case, information will be available about the jobs that ran, but not the details of the files that were backed up. Normally, when a Job record is purged, all its File records will also be purged.
The Volume Retention Period is the minimum of time that a Volume will be kept before it is reused. Note, if all the Jobs and Files associated to a Volume are pruned from Catalog, Bacula may reuse this Volume before its retention time. Bacula will normally never overwrite a Volume that contains the only backup copy of a file. Under ideal conditions, the Catalog would retain entries for all files backed up for all current Volumes. Once a Volume is overwritten, the files that were backed up on that Volume are automatically removed from the Catalog. However, if there is a very large pool of Volumes or a Volume is never overwritten, the Catalog database may become enormous. To keep the Catalog to a manageable size, the backup information should be removed from the Catalog after the defined File Retention Period. Bacula provides the mechanisms for the catalog to be automatically pruned according to the retention periods defined.
Bacula is a backup, restore and verification program and is not a complete disaster recovery system in itself, but it can be a key part of one if you plan carefully and follow the instructions included in the Disaster Recovery Chapter of this manual.
With proper planning, as mentioned in the Disaster Recovery chapter, Bacula can be a central component of your disaster recovery system. For example, if you have created an emergency boot disk, and/or a Bacula Rescue disk to save the current partitioning information of your hard disk, and maintain a complete Bacula backup, it is possible to completely recover your system from _left_quote_bare metal_right_quote_ that is starting from an empty disk.
If you have used the WriteBootstrap record in your job or some other means to save a valid bootstrap file, you will be able to use it to extract the necessary files (without using the catalog or manually searching for the files to restore).
The block diagram (here) shows the typical interactions between the Bacula Services for a backup job. Each block represents in general a separate process (normally a daemon). In general, the Director oversees the flow of information. It also maintains the Catalog.
The password information that might be present in some specific plugin FileSets is now hidden from the status client console output.
New security events are generated after incorrect network connections.
The restricted console have now a better control over various bconsole commands such last list, restore, ...
The Bacula Storage Daemon can now encrypt the data at the volume level to enhance security of data at rest. The volumes cannot be read by a system that doesn't have the correct encryption keys.
More information can be found in the Security chapter of the documentation (here).
Bacula allows you to configure your jobs to detect known Malware. The detection can be done at the end of the Backup job and/or with a Verify job. The Malware database can be downloaded from different providers, the default is set to abuse.ch. If a Backup job detects a malware in the backup content, an error is reported and the Job status is adapted.
20-Sep 12:26 zog8-dir JobId 9: Start Backup JobId 9, Job=backup.2022-09-20_12.26.30_13 ... 20-Sep 12:26 zog8-dir JobId 9: [DI0002] Checking file metadata for Malwares 20-Sep 12:26 zog8-dir JobId 9: Error: [DE0007] Found Malware(s) on JobIds 9 Build OS: x86_64-pc-linux-gnu archlinux JobId: 9 Job: backup.2022-09-20_12.26.30_13 Backup Level: Full ... Last Volume Bytes: 659,912,644 (659.9 MB) Non-fatal FD errors: 1 SD Errors: 0 FD termination status: OK SD termination status: OK Termination: Backup OK -- with warnings
The list of the Malware detected in a given Job can be displayed with the list files type=malware command.
*list files type=malware jobid=1 +-------+-----------------------------+---------------+----------+ | jobid | filename | description | source | +-------+-----------------------------+---------------+----------+ | 1 | /tmp/regress/build/po/fr.po | Malware found | abuse.ch | +-------+-----------------------------+---------------+----------+
See the (here) section of this manual for more information.
The TOTP (Time based One Time Password) Authentication Plugin is compatible with the RFC 6238. Many smartphone apps are available to store the keys and compute the TOTP code.
The standard password and encryption mechanisms will still be used to accept an incoming console connection. Once accepted, the Console will prompt for a second level of authentication with a TOTP secret key generated from a shared token.
To enable this feature, you needed to install the bacula-totp-dir-plugin package on your Director system, then to set the PluginDirectory directive of the Director resource and configure the AuthenticationPlugin directive of a given restricted Console in the Director configuration file.
# in bacula-dir.conf Director { Name = myname-dir ... Plugin Directory = /opt/bacula/plugins } Console { Name = "totpconsole" Password = "xxx" Authentication Plugin = "totp" }
The matching Console configuration in bconsole.conf has no extra settings compared to a standard restricted Console.
# in bconsole.conf Console { Name = totpconsole Password = "xxx" # Same as in bacula-dir.conf/Console } Director { Name = mydir-dir Address = localhost Password = notused }
At the first console connection, if the TLS link is correctly setup (using the shared secret key), the plugin will generate a specific random key for the console and display a QR code in the console output. The user must then scan the QR code with his smartphone using an app such as Aegis (Opensource) or Google Authenticator. The plugin can also be configured to send the QR code via an extranal program.
More information can be found in the main manual on (here).
The StorageGroup feature is now compatible with Copy and Migration jobs.
This policy queries each Storage Daemon in the list for its FreeSpace (as a sum of devices specified in the Storage Daemon config) and sort the list by FreeSpace returned.
This policy ensures that job is backed up to the storage where the same job (of the same level, i.e. Full or Incremental) has been backed up the longest time ago. The goal is to split the jobs to improve redundancy.
This policy ensures that job is backed up on the storage where the same job (with same level i.e. Full or Incremental) has been backed up to the longest time ago. The goal is to spread the jobs to improve redundancy.
This policy ensures that a job is backed up to the storage with the most free
space and least running jobs. Within the candidates storages, the least used
one will be selected. Candidate storages are determined by the
StorageGroupPolicyThreshold Job directive. If MaxFreeSpace is the largest
amount of free space for all storages in the group, a storage will be a
candidate if its free space is above .
For example:
with StorageGroupPolicyThreshold=100MB and storages free space being: Storage1 = 500GB free Storage2 = 200GB free Storage3 = 400GB free storage4 = 500GB free In this case MaxFreeSpace=500GB. Storage 1, 4 and 3 are candidates. If 5 jobs are running on Storage1, 2 on Storage4, and 3 on Storage3 then Storage4 will be the selected storage.
Storage Groups can be used as follows (as part of Job and Pool configuration):
Job { ... Storage = File1, File2, File3 ... } Pool { ... Storage = File4, File5, File6 StorageGroupPolicy = FreeSpaceLeastUsed StorageGroupPolicyThreshold = 200 MB ... }
New FileDaemon directives lets the deamon control which client directories are allowed for backup on a per-director basis. Directives can be specified as comma-separated list of directories. The simplest version of the AllowedBackupDirectories and ExcludedBackupDirectories directives may look as follows:
# in bacula-fd.conf Director { Name = myname-dir ... AllowedBackupDirectories = "/path/to/allowed/directory" } Director { Name = my-other-dir ... ExcludedBackupDirectories = "/path/to/excluded/directory" }
This directive works on the FD side, and is fully independent of the include/exclude part of the Fileset defined in the Director's config file. Nothing is backed up if none of the files defined in the Fileset is inside FD's allowed directory.
# in bacula-fd.conf Director { Name = myname-dir ... AllowedRestoreDirectories = "/path/to/directory" }
# in bacula-fd.conf Director { Name = myname-dir ... AllowedScriptDirectories = "/path/to/directory" }
When this directive is set, Bacula is also checking programs to be run against set of not-allowed characters. When following resource:
FileSet { Name = "Fileset_1" Include { File = "\\|/path/to/binary &" } }is defined inside the Director's config file, Bacula won't backup any file for such Fileset. It's because of the '&' character, which is not allowed when the 'Allowed Script Directories' is used on the Client's side. Full list of not-allowed characters:
$ ! ; \ & < > ` ( )
To disable all command sent by the Director, it is possible to use the following configuration in your FileDaemon:
AllowedScriptDirectories = none
The FileDaemon Antivirus plugin provides integration between the ClamAV Antivirus daemon and Bacula Verify Jobs, allowing post-backup virus detection within Bacula Enterprise.
More information can be found in the Antivirus Plugin user's guide.
This feature can only be used if Bacula is run as a systemd service because only then, with proper capabilities set for the daemon, it's allowed to manage Volume Files' attributes. The `show storage` bconsole command informs if Bacula has needed capabilities.
For File-based volumes Bacula will set the Append Only attribute during the first backup job that uses the new volume. Note that the flag is set when Bacula actually uses volume for the first time, not when the volume is labelled (either automatically or using the `label` command). This will prevent Volumes to lost data by being ovewritten.
The Append Only file attribute is cleared when the volume is being relabeled.
Bacula is now also able to set the Immutable file attribute on a file volume which is marked as Full. Note that as of now, Immutable flag is set only for Full volumes, setting it for other statuses (e.g. 'Used' volumes) may come in the future releases.
When a volume is Full and has the Immutable flag set, it cannot be relabeled and reused until the expiration period elapses. This helps to protect volumes from being reused too early, according to the protection period set.
If Volume's filesystem does not support the Append only or Immutable flags, a proper warning message is printed in the job log and Bacula proceeds with the usual backup workflow.
There are three new directives available to set on a per-device basis to control the the Volume Protection behavior:
Note: Both Append Only and Immutable flags set for volumes cannot be modified using catalog-related commands, e.g. purging a volume won't clear the Immutable flag, that can only be done when MinimumVolumeProtection time expires.
If administrator wants to manually remove file attributes, chattr must be used:
In some cases, for example when the status of the Volume is changed by the Director via the update volume command, the Storage Daemon will not be able to change the permission on the Volume. Some Volumes may have the Full/Used status without the proper protection.
The command update volumeprotect is designed to determine the list of the volumes that are not protected and connect the Storage Daemon to update the permissions. It can be executed in an Admin job once a day.
*update Update choice: 1: Volume parameters 2: Pool from resource 3: Slots from autochanger 4: Long term statistics 5: Snapshot parameters 6: Volume protection attributes on Storage Daemon Choose catalog item to update (1-6): 6 Found 1 volumes with status Used/Full that must be protected Connected to Storage "File2" at zog8:8103 with TLS 3000 Marking volume "Vol-0009" as read-only.
or via update volumeprotect
*update volumeprotect Found 1 volumes with status Used/Full that must be protected Connected to Storage "File2" at zog8:8103 with TLS 3000 Marking volume "Vol-0009" as read-only.
The command can be scheduled in an Admin job
Job { Name = adm-update-protected Type = Admin Runscript { Console = "update volumeprotect" RunsOnClient = no RunsWhen = Before } JobDefs = DefaultJob }
The ZSTD compression algorithm is now available in the FileSet option directive Compression. It is possible to configure ZSTD level 1 zstd1, level 10 zstd10 and level 19 zstd19. The default zstd compression is 10.
The Bacula Kubernetes plugin will save all the important Kubernetes resources which make up the application or service. The plugin can now use the CSI Snapshot method to backup persistent volumes.
A new Amazon Cloud Driver is available for beta testing. In the long term, it will enhance and replace the existing S3 cloud driver. The aws tool provided by Amazon is needed to use this cloud driver. The Amazon cloud driver is available within the bacula-cloud-storage-s3 package and can be enabled by using _left_quote_Driver = Amazon_right_quote_ in the Cloud Storage Daemon resource.
Bacula version 15.0 uses a new volume version named BB03. The new format adds the support for Volume Encryption, and the previous 32bits CRC32 checksum was replaced by the faster 64bits XXH64.
Volumes written with the BB03 format can only be read by Bacula version 15 or later. Old BB02 volumes can still be restored, and Volumes may start with BB02 blocks, and continue with BB03 blocks.
It is not possible to use the Volume Encryption=yes directive on a volume that was labeled using the BB02 format. In that case, the volume will be automatically marked as Used.
The status director command can now report the progress of Copy and Migration Jobs.
The status director bconsole command has been updated to limit the number of scheduled jobs listed by default to 50.
The keyword limit can be used to choose how many lines will be printed.
* status director limit=5 127.0.0.1-dir Version: 14.1.5 (26 October 2022) x86_64-pc-linux-gnu archlinux Daemon started 31-Oct-22 17:27, conf reloaded 31-Oct-2022 17:27:55 Jobs: run=2, running=0 max=4 mode=1,2010 Crypto: fips=no crypto=OpenSSL 1.0.2u 20 Dec 2019 Heap: heap=675,840 smbytes=774,259 max_bytes=2,058,994 bufs=2,470 max_bufs=4,024 Res: njobs=24 nclients=1 nstores=3 npools=1 ncats=1 nfsets=25 nscheds=3 Plugin: ldap totp Scheduled Jobs (5/1440): Level Type Pri Scheduled Job Name Volume =================================================================================== Full Backup 10 31-Oct-22 17:29 NightlySave0 TestVolume001 Full Backup 10 31-Oct-22 17:30 NightlySave1 TestVolume001 Full Backup 10 31-Oct-22 17:31 NightlySave2 TestVolume001 Full Backup 10 31-Oct-22 17:32 NightlySave3 TestVolume001 Full Backup 10 31-Oct-22 17:33 NightlySave4 TestVolume001 5 scheduled Jobs over 1440 are displayed. Use the limit parameter to display more Jobs. ==== Running Jobs: Console connected using TLS at 31-Oct-22 17:27 No Jobs running. ==== Terminated Jobs: JobId Level Files Bytes Status Finished Name ==================================================================== 1 Full 35 5.070 M OK 31-Oct-22 15:50 NightlySave 2 Full 35 5.070 M OK 31-Oct-22 17:27 NightlySave 3 Full 35 5.070 M OK 31-Oct-22 17:28 NightlySave ====
The APIv2 json output support has been added for the status dir scheduled command.
FileSet Content Overview
Now, the Catalog stores an overview of the FileSet definition in the **Content** field. If the FileSet handles files and directories, the Content field will be set to **files**. If any plugins are used, each plugin will be inserted into the Content field.
*sql SELECT Content FROM FileSet; +------------------------------+ | content | +------------------------------+ | bpipe | +------------------------------+
New Job Attributes
New SQL attributes have been added to the Job table such as **isVirtualFull**, **Encrypted**, **LastReadStorageId**, **WriteStorageId**, **Rate**, **CompressRatio**, **StatusInfo**, and so on.
New Media Attributes
New SQL attributes have been added to the Media table such as **UseProtect**, **Protected**, **VolEncrypted**
Bconsole has a new .search command to search into the catalog accross client, job and volumes.
* .search text=sometext volume= client=sometext-fd job=
The text to be searched must have 4 characters at the minimum.
The Job RunScript feature has been enhanced to control the start of a Job inside the Run Queue. When a Job is starting, the Director controls that resources are available for the Job to start properly, if these resources are not available, the Job will stay in the queue, waiting to acquire them.
It is now possible to execute a script to control any kind of external and custom resources and decide when a Job should start. For example, a script might control the load average of a server before to start a Job to find the best execution time.
More information can be found in the Director documentation (here).
The Director PID file timestamp is now updated after a successful reload command. External tools can use this information to perform certain actions such as to clear the cache.
Job { ... Runscript { RunsOnClient = no RunsWhen = AtJobCompletion Command = "mail command" AbortJobOnError = yes } }It has been added because the RunsWhen keyword After was not designed to update the job status if the command fails.
The console has been improved to support a JSON output to list catalog objects and various daemon output. The new _left_quote_.jlist_right_quote_ command is a shortcut of the standard _left_quote_list_right_quote_ command and will display the results in a JSON table. All options and filters of the _left_quote_list_right_quote_ command can be used in the _left_quote_.jlist_right_quote_ command. Only catalog objects are listed with the _left_quote_list_right_quote_ or _left_quote_.jlist_right_quote_ commands. Resources such as Schedule, FileSets, etc... are not handled by the _left_quote_list_right_quote_ command.
See the _left_quote_help list_right_quote_ bconsole output for more information about the _left_quote_list_right_quote_ command. The Bacula configuration can be displayed in JSON format with the standard _left_quote_bdirjson_right_quote_, _left_quote_bsdjson_right_quote_, _left_quote_bfdjson_right_quote_ and _left_quote_bbconsjson_right_quote_ tools.
*.jlist jobs {"type": "jobs", "data":[{"jobid": 1,"job": "CopyJobSave.2021-10-04_18.35.55_03",...
*.api 2 api_opts=j *.status dir header {"header":{"name":"127.0.0.1-dir","version":"12.8.2 (09 September 2021)"...