next up previous contents index
Next: Release Version 5.0.3 Up: Bacula Main Reference Previous: Enterprise Bacula New Features   Contents   Index

Subsections

Community Bacula New Features in 5.1.x

This chapter presents the new features that have been added to the current Community version of Bacula that is under development. This version will be released at some later date, probably near the end of April 2011.

There are additional features (plugins) available in the Enterprise version that are described in another chapter. A subscription to Bacula Systems is required for the Enterprise version.

LZO Compression

LZO compression was added in the Unix File Daemon. From the user point of view, it works like the GZIP compression (just replace compression=GZIP with compression=LZO).

For example:

Include {
   Options { compression=LZO }
   File = /home
   File = /data
}

LZO provides much faster compression and decompression speed but lower compression ratio than GZIP. It is a good option when you backup to disk. For tape, the built-in compression may be a better option.

LZO is a good altenative for GZIP1 when you don't want to slow down your backup. On a modern CPU it should be able to run almost as fast as:

Note that bacula only use one compression level LZO1X-1.


The code for this feature was contributed by Laurent Papier.

New Tray Monitor

Since the old integrated Windows tray monitor doesn't work anymore with recent Windows versions, we have written a new Qt Tray Monitor that is available for both Linux and Windows. This new version allows you to run Backup from the tray monitor menu.

Figure 3.1: New tray monitor
\includegraphics[width=10cm]{tray-monitor}

Figure 3.2: Run a Job through the new tray monitor
\includegraphics[width=10cm]{tray-monitor1}

To be able to run a job from the tray monitor, you need to allow specific commands in the Director monitor console:

Console {
    Name = win2003-mon
    Password = "xxx"
    CommandACL = status, .clients, .jobs, .pools, .storage, .filesets, .messages, run
    ClientACL = *all*               # you can restrict to a specific host
    CatalogACL = *all*
    JobACL = *all*
    StorageACL = *all*
    ScheduleACL = *all*
    PoolACL = *all*
    FileSetACL = *all*
    WhereACL = *all*
}


This project was funded by Bacula Systems and is available with Bacula the Enterprise Edition and the Community Edition.

Purge Migration Job

The new Purge Migration Job directive may be added to the Migration Job definition in the Director's configuration file. When it is enabled the Job that was migrated during a migration will be purged at the end of the migration job.

For example:

Job {
  Name = "migrate-job"
  Type = Migrate
  Level = Full
  Client = localhost-fd
  FileSet = "Full Set"
  Messages = Standard
  Storage = DiskChanger
  Pool = Default
  Selection Type = Job
  Selection Pattern = ".*Save"
...
  Purge Migrated Job = yes
}


This project was submited by Dunlap Blake, testing and documentation was funded by Bacula Systems.

Changes in Bvfs (Bacula Virtual FileSystem)

Bat has now a bRestore panel that uses Bvfs to display files and directories. It's still experimental.

Figure 3.3: Bat Brestore Panel
\includegraphics[width=12cm]{bat-brestore}

Important, the Bvfs module is not currently compatible with BaseJobs, Copy and Migration jobs.


This project was funded by Bacula Systems.

General notes

Get dependent jobs from a given JobId

Bvfs allows you to query the catalog against any combination of jobs. You can combine all Jobs and all FileSet for a Client in a single session.

To get all JobId needed to restore a particular job, you can use the .bvfs_get_jobids command.

.bvfs_get_jobids jobid=num [all]

.bvfs_get_jobids jobid=10
1,2,5,10
.bvfs_get_jobids jobid=10 all
1,2,3,5,10

In this example, a normal restore will need to use JobIds 1,2,5,10 to compute a complete restore of the system.

With the all option, the Director will use all defined FileSet for this client.

Generating Bvfs cache

The .bvfs_update command computes the directory cache for jobs specified in argument, or for all jobs if unspecified.

.bvfs_update [jobid=numlist]

Example:

.bvfs_update jobid=1,2,3

You can run the cache update process in a RunScript after the catalog backup.

Get all versions of a specific file

Bvfs allows you to find all versions of a specific file for a given Client with the .bvfs_version command. To avoid problems with encoding, this function uses only PathId and FilenameId. The jobid argument is mandatory but unused.

.bvfs_versions client=filedaemon pathid=num filenameid=num jobid=1
PathId FilenameId FileId JobId LStat Md5 VolName Inchanger
PathId FilenameId FileId JobId LStat Md5 VolName Inchanger
...

Example:

.bvfs_versions client=localhost-fd pathid=1 fnid=47 jobid=1
1  47  52  12  gD HRid IGk D Po Po A P BAA I A   /uPgWaxMgKZlnMti7LChyA  Vol1  1

List directories

Bvfs allows you to list directories in a specific path.

.bvfs_lsdirs pathid=num path=/apath jobid=numlist limit=num offset=num
PathId  FilenameId  FileId  JobId  LStat  Path
PathId  FilenameId  FileId  JobId  LStat  Path
PathId  FilenameId  FileId  JobId  LStat  Path
...

You need to pathid or path. Using path="" will list ``/'' on Unix and all drives on Windows. If FilenameId is 0, the record listed is a directory.

.bvfs_lsdirs pathid=4 jobid=1,11,12
4       0       0       0       A A A A A A A A A A A A A A     .
5       0       0       0       A A A A A A A A A A A A A A     ..
3       0       0       0       A A A A A A A A A A A A A A     regress/

In this example, to list directories present in regress/, you can use

.bvfs_lsdirs pathid=3 jobid=1,11,12
3       0       0       0       A A A A A A A A A A A A A A     .
4       0       0       0       A A A A A A A A A A A A A A     ..
2       0       0       0       A A A A A A A A A A A A A A     tmp/

List files

Bvfs allows you to list files in a specific path.

.bvfs_lsfiles pathid=num path=/apath jobid=numlist limit=num offset=num
PathId  FilenameId  FileId  JobId  LStat  Path
PathId  FilenameId  FileId  JobId  LStat  Path
PathId  FilenameId  FileId  JobId  LStat  Path
...

You need to pathid or path. Using path="" will list ``/'' on Unix and all drives on Windows. If FilenameId is 0, the record listed is a directory.

.bvfs_lsfiles pathid=4 jobid=1,11,12
4       0       0       0       A A A A A A A A A A A A A A     .
5       0       0       0       A A A A A A A A A A A A A A     ..
1       0       0       0       A A A A A A A A A A A A A A     regress/

In this example, to list files present in regress/, you can use

.bvfs_lsfiles pathid=1 jobid=1,11,12
1   47   52   12    gD HRid IGk BAA I BMqcPH BMqcPE BMqe+t A     titi
1   49   53   12    gD HRid IGk BAA I BMqe/K BMqcPE BMqe+t B     toto
1   48   54   12    gD HRie IGk BAA I BMqcPH BMqcPE BMqe+3 A     tutu
1   45   55   12    gD HRid IGk BAA I BMqe/K BMqcPE BMqe+t B     ficheriro1.txt
1   46   56   12    gD HRie IGk BAA I BMqe/K BMqcPE BMqe+3 D     ficheriro2.txt

Restore set of files

Bvfs allows you to create a SQL table that contains files that you want to restore. This table can be provided to a restore command with the file option.

.bvfs_restore fileid=numlist dirid=numlist hardlink=numlist path=b2num
OK
restore file=?b2num ...

To include a directory (with dirid), Bvfs needs to run a query to select all files. This query could be time consuming.

hardlink list is always composed of a serie of two numbers (jobid, fileindex). This information can be found in the LinkFI field of the LStat packet.

The path argument represents the name of the table that Bvfs will store results. The format of this table is b2[0-9]+. (Should start by b2 and followed by digits).

Example:

.bvfs_restore fileid=1,2,3,4 hardlink=10,15,10,20 jobid=10 path=b20001
OK

Cleanup after restore

To drop the table used by the restore command, you can use the .bvfs_cleanup command.

.bvfs_cleanup path=b20001

Changes in the pruning algorithm

We rewrote the job pruning algorithm in this version. Previously, in some users reported that the pruning process at the end of jobs was very long. It should not be longer the case. Now, Bacula won't prune automatically a Job if this particular Job is needed to restore data. Example:

JobId: 1  Level: Full
JobId: 2  Level: Incremental
JobId: 3  Level: Incremental
JobId: 4  Level: Differential
.. Other incrementals up to now

In this example, if the Job Retention defined in the Pool or in the Client resource causes that Jobs with Jobid in 1,2,3,4 can be pruned, Bacula will detect that JobId 1 and 4 are essential to restore data at the current state and will prune only JobId 2 and 3.

Important, this change affect only the automatic pruning step after a Job and the prune jobs Bconsole command. If a volume expires after the VolumeRetention period, important jobs can be pruned.

Ability to Verify any specified Job

You now have the ability to tell Bacula which Job should verify instead of automatically verify just the last one.

This feature can be used with VolumeToCatalog, DiskToCatalog and Catalog level.

To verify a given job, just specify the Job jobid in argument when starting the job.

*run job=VerifyVolume jobid=1 level=VolumeToCatalog
Run Verify job
JobName:     VerifyVolume
Level:       VolumeToCatalog
Client:      127.0.0.1-fd
FileSet:     Full Set
Pool:        Default (From Job resource)
Storage:     File (From Job resource)
Verify Job:  VerifyVol.2010-09-08_14.17.17_03
Verify List: /tmp/regress/working/VerifyVol.bsr
When:        2010-09-08 14:17:31
Priority:    10
OK to run? (yes/mod/no):


This project was funded by Bacula Systems and is available with Bacula Enterprise Edition and Community Edition.

Additions to RunScript variables

You can have access to JobBytes and JobFiles using %b and %F in your runscript command. The Client address is now available through %h.

RunAfterJob = "/bin/echo Job=%j JobBytes=%b JobFiles=%F ClientAddress=%h"

Additions to the Plugin API

The bfuncs structure has been extended to include a number of new entrypoints.

bfuncs

The bFuncs structure defines the callback entry points within Bacula that the plugin can use register events, get Bacula values, set Bacula values, and send messages to the Job output or debug output.

The exact definition as of this writing is:

typedef struct s_baculaFuncs {
   uint32_t size;
   uint32_t version;
   bRC (*registerBaculaEvents)(bpContext *ctx, ...);
   bRC (*getBaculaValue)(bpContext *ctx, bVariable var, void *value);
   bRC (*setBaculaValue)(bpContext *ctx, bVariable var, void *value);
   bRC (*JobMessage)(bpContext *ctx, const char *file, int line,
       int type, utime_t mtime, const char *fmt, ...);
   bRC (*DebugMessage)(bpContext *ctx, const char *file, int line,
       int level, const char *fmt, ...);
   void *(*baculaMalloc)(bpContext *ctx, const char *file, int line,
       size_t size);
   void (*baculaFree)(bpContext *ctx, const char *file, int line, void *mem);
   
   /* New functions follow */
   bRC (*AddExclude)(bpContext *ctx, const char *file);
   bRC (*AddInclude)(bpContext *ctx, const char *file);
   bRC (*AddIncludeOptions)(bpContext *ctx, const char *opts);
   bRC (*AddRegex)(bpContext *ctx, const char *item, int type);
   bRC (*AddWild)(bpContext *ctx, const char *item, int type);
   bRC (*checkChanges)(bpContext *ctx, struct save_pkt *sp);

} bFuncs;

AddExclude
can be called to exclude a file. The file string passed may include wildcards that will be interpreted by the fnmatch subroutine. This function can be called multiple times, and each time the file specified will be added to the list of files to be excluded. Note, this function only permits adding excludes of specific file or directory names, or files matched by the rather simple fnmatch mechanism. See below for information on doing wild-card and regex excludes.

NewInclude
can be called to create a new Include block. This block will be added before any user defined Include blocks. This function can be called multiple times, but each time, it will create a new Include section (not normally needed). This function should be called only if you want to add an entirely new Include block.

AddInclude
can be called to add new files/directories to be included. They are added to the current Include block. If NewInclude has not been included, the current Include block is the last one that the user created. This function should be used only if you want to add totally new files/directories to be included in the backup.

NewOptions
adds a new Options block to the current Include in front of any other Options blocks. This permits the plugin to add exclude directives (wild-cards and regexes) in front of the user Options, and thus prevent certain files from being backed up. This can be useful if the plugin backs up files, and they should not be also backed up by the main Bacula code. This function may be called multiple times, and each time, it creates a new prepended Options block. Note: normally you want to call this entry point prior to calling AddOptions, AddRegex, or AddWild.

AddOptions
allows the plugin it set options in the current Options block, which is normally created with the NewOptions call just prior to adding Include Options. The permitted options are passed as a character string, where each character has a specific meaning as defined below:

a
always replace files (default).
e
exclude rather than include.
h
no recursion into subdirectories.
H
do not handle hard links.
i
ignore case in wildcard and regex matches.
M
compute an MD5 sum.
p
use a portable data format on Windows (not recommended).
R
backup resource forks and Findr Info.
r
read from a fifo
S1
compute an SHA1 sum.
S2
compute an SHA256 sum.
S3
comput an SHA512 sum.
s
handle sparse files.
m
use st_mtime only for file differences.
k
restore the st_atime after accessing a file.
A
enable ACL backup.
Vxxx:
specify verify options. Must terminate with :
Cxxx:
specify accurate options. Must terminate with :
Jxxx:
specify base job Options. Must terminate with :
Pnnn:
specify integer nnn paths to strip. Must terminate with :
w
if newer
Zn
specify gzip compression level n.
K
do not use st_atime in backup decision.
c
check if file changed during backup.
N
honor no dump flag.
X
enable backup of extended attributes.

AddRegex
adds a regex expression to the current Options block. The fillowing options are permitted:
(a blank) regex applies to whole path and filename.
F
regex applies only to the filename (directory or path stripped).
D
regex applies only to the directory (path) part of the name.

AddWild
adds a wildcard expression to the current Options block. The fillowing options are permitted:
(a blank) regex applies to whole path and filename.
F
regex applies only to the filename (directory or path stripped).
D
regex applies only to the directory (path) part of the name.

checkChanges
call the check_changes() function in Bacula code that can use Accurate code to compare the file information in argument with the previous file information. The delta_seq attribute of the save_pkt will be updated, and the call will return bRC_Seen if the core code wouldn't decide to backup it.

Bacula events

The list of events has been extended to include:

typedef enum {
  bEventJobStart        = 1,
  bEventJobEnd          = 2,
  bEventStartBackupJob  = 3,
  bEventEndBackupJob    = 4,
  bEventStartRestoreJob = 5,
  bEventEndRestoreJob   = 6,
  bEventStartVerifyJob  = 7,
  bEventEndVerifyJob    = 8,
  bEventBackupCommand   = 9,
  bEventRestoreCommand  = 10,
  bEventLevel           = 11,
  bEventSince           = 12,
   
  /* New events */
  bEventCancelCommand                   = 13,
  bEventVssBackupAddComponents          = 14,
  bEventVssRestoreLoadComponentMetadata = 15,
  bEventVssRestoreSetComponentsSelected = 16,
  bEventRestoreObject                   = 17,
  bEventEndFileSet                      = 18,
  bEventPluginCommand                   = 19,
  bEventVssBeforeCloseRestore           = 20,
  bEventVssPrepareSnapshot              = 21

} bEventType;

bEventCancelCommand
is called whenever the currently running Job is cancelled */

bEventVssBackupAddComponents

bEventVssPrepareSnapshot
is called before creating VSS snapshots, it provides a char[27] table where the plugin can add Windows drives that will be used during the Job. You need to add them without duplicates, and you can use in fd_common.h add_drive() and copy_drives() for this purpose.

ACL enhancements

The following enhancements are made to the Bacula Filed with regards to Access Control Lists (ACLs)


This project was funded by Planets Communications B.V. and ELM Consultancy B.V. and is available with Bacula Enterprise Edition and Community Edition.

XATTR enhancements

The following enhancements are made to the Bacula Filed with regards to Extended Attributes (XATTRs)


This project was funded by Planets Communications B.V. and ELM Consultancy B.V. and is available with Bacula Enterprise Edition and Community Edition.

Class based database backend drivers

All current database backend drivers for catalog information are rewritten to use a set of multi inherited C++ classes which abstract the specific database specific internals and make sure we have a more stable generic interface with the rest of sql code. From now on there is a strict boundery between the sql code and the low-level database functions. This new interface should also make it easier to add a new backend for a currently unsupported database. As part of the rewrite the SQLITE2 code was removed (e.g. only SQLITE3 is now supported). An extra bonus of the new code is that you can configure multiple backends in the configure and build all backends in one compile session and select the correct database backend at install time. This should make it a lot easier for packages maintainers.


We also added cursor support for PostgreSQL backend, this improves memory usage for large installation.


This project was funded by Planets Communications B.V. and ELM Consultancy B.V. and Bacula Systems and is available with Bacula Enterprise Edition and Community Edition.

Hash list enhancements

The htable hash table class has been extended with extra hash functions for handling next to char pointer hashes also 32 bits and 64 bits hash keys. Also the hash table initialization routines have been enhanced with support for passing a hint as to the number of initial pages to use for the size of the hash table. Until now the hash table always used a fixed value of 10 Mb. The private hash functions of the mountpoint entry cache have been rewritten to use the new htable class with a small memory footprint.


This project was funded by Planets Communications B.V. and ELM Consultancy B.V. and Bacula Systems and is available with Bacula Enterprise Edition and Community Edition.


next up previous contents index
Next: Release Version 5.0.3 Up: Bacula Main Reference Previous: Enterprise Bacula New Features   Contents   Index
Eric 2011-08-08