Features Overview
- Online backup based on transactions of any running QEMU VMs
- Full and incremental virtual disk image backup at the block level
- Easy VM recovery to the original or alternative cluster
- Capability to restore virtual disk images as QCOW2 files
- Capability to fully restore a guest VM, including VM configurations
- Capability to fully restore destination disk images for genuine QEMU
- Capability to restore VM virtual disk images to an alternate directory
Image Backup with QEMU Plugin
With the image-level backup strategy, the Bacula Enterprise QEMU Plugin will save VM disks as QCOW2 images for QEMU VMs, both for full and incremental backups.
For this to work, a Bacula File Daemon is not required in each guest VM. The Bacula QEMU Plugin will contact the QEMU hypervisor to read and save the content of virtual machine disks using the QMP transaction backup function and will dump them using the QMP API.
Bacula doesn’t need to traverse the client’s file systems to initiate, open, read, and close files, so it consumes fewer resources in the QEMU infrastructure than file-level backup in each VM would. On the other hand, Bacula will also read and save unnecessary data in VMs, such as swap or temporary files.
Backup and Restoration Overview
Backup
The QEMU Plugin supports the QMP control protocol, which is a low-level management interface for all flavors of QEMU hypervisors. In most cases, this interface is used by hypervisor management tools to manage QEMU guest VMs.
The current version of the Plugin has been tested on a few types of QEMU hypervisors:
- Genuine QEMU – mode=QEMU (default)
- Proxmox (PVE) – mode=PVE
- Oracle Virtualization
- KVM
The core procedure of backup and restoration is the same for each type of QEMU hypervisor. The main difference is how a VM is selected to be restored and how the restoration process is finalized.
The central backup of a single guest VM consists of the following steps:
- Query VM information and save it
- For a full backup level, during the backup transaction, create or clear the dirty block and dump the VM disk image to the workspace (check Working Directory Location)
- For an incremental backup level, during the backup transaction, dump the incremental VM disk image to the workspace and clear the dirty bitmap
- Once the VM disk image dump is ready, the Bacula FileDaemon will send the data to the Storage Daemon.
Backups can only be performed for running VMs. Any initial full backup will create a backup chain used for subsequent incremental backups. The current version of the plugin creates and maintains a single backup chain for each VM.
Warning
You should never mix different backup jobs for a single QEMU VM, as this will break the backup chain and backups will become unrecoverable!
The QEMU Plugin will log the start and end of each guest VM backup:
JobId 104: Start Backup JobId 104, Job=proxmox.2021-10-28_12.04.54_42 JobId 104: Recycled volume "vol01" JobId 104: Using Device "FileChgr1-Dev2" to write. JobId 104: qemu: Connected to Proxmox pve-manager/6.4-13/9f411e79 (running kernel: 5.4.143-1-pve) JobId 104: qemu: Start Backup vm: vm1 (100) JobId 104: qemu: Finish backup of vm1/drive-scsi0 ...
The backup will create a single file (.qcow2) for each saved VM disk device and a single VM configuration file (config.json) during a full backup.
/@qemu/<vm-name>/<vmid>/<drive-name>.qcow2 for each VM disk image /@qemu/<vm-name>/<vmid>/config.json for VM configuration data
Tip
For genuine QEMU mode, <vmid> is generally set to zero, and <vm-name> is the value defined by the QEMU -name execution parameter in the QEMU command line.
Several files will be created during a backup if multiple guest VMs are backed up with a job (only in Proxmox backup mode). The distinct file names, as shown above, will help locate the appropriate guest images for restoration.
Restore
The QEMU Plugin allows the following destinations for restore operations:
- Restoration to a local directory as QCOW2 disk image files and a single configuration file
- Restoration to a genuine QEMU hypervisor at the original device file location
- Restoration to the Proxmox virtualization infrastructure as an original or new VMID
- Restoration to a Local Directory
To use this mode, the where=/some_directory parameter of a Bacula restoration is set as a complete path on the server where the QEMU Plugin is installed. If the path doesn’t exist, the Bacula QEMU Plugin will create it. Bacula will automatically apply incremental backups to the target disks. The end result will be a QCOW2 disk image file “ready for use”. For this type of destination, you will always get disk images in the QCOW2 format.
Restore to Genuine QEMU
To use this restoration mode, the where=/ parameter of a Bacula restoration is used. The VM guest disk images will be restored and automatically patched with all incremental backups in the workspace (check Working Directory Location for details). Then, all patched and ready-to-use image files will be converted back to the original destination location and image format when the restoration plugin parameter convdestination: yes is set. This allows obtaining QEMU guest VM images ready to run in their original location.
This restoration mode is selected for all backups performed for Genuine QEMU.
Restore to Proxmox
To use this restoration mode, the where=/ parameter of a Bacula restore is used. The VM disk images will be restored and automatically patched with all incremental backups in the workspace (check Working Directory Location for details). Then, all patched and ready-to-use image files will be transferred to the Proxmox storage connected to the original image.
The QEMU Plugin will attempt to create the same VMID that was present during backup. If the guest VM already exists with that VMID, the plugin will allocate a new VMID. It will never overwrite an existing guest VM.
All other guest VM configuration parameters will be restored as they were copied, including network MAC addresses. For this reason, it’s recommended to inspect and possibly update the VM configuration before starting it. Otherwise, resource conflicts may arise.
The target storage to be used for the restored guest VM disk(s) can be defined using the storage restoration option of the plugin. If this option is not set, all VM disks will be restored to their original storage.
To list available storages, a listing mode is available, as described in the listing chapter.
This restoration mode is selected for all backups performed for Proxmox.
Attention
The QEMU Plugin cannot be used for automatic migration of QEMU guest VMs from Genuine QEMU to a Proxmox virtualization infrastructure, but you can perform a migration from Proxmox to Genuine QEMU using the backup=QEMU mode for both.
Installation
The Bacula Enterprise File Daemon and its QEMU Plugin need to be installed on the Proxmox hypervisor host running the guest VMs that will be copied. Proxmox uses a customized Debian distribution, so the Bacula Enterprise File Daemon for this platform should be used.
The installation of the File Daemon and Bacula Enterprise QEMU Plugin is most easily performed using the bee_installation_manager: https://www.bacula.lat/automatic-server-installation-with-bacula-enterprise-manager-centos-debian-etc/?lang=en
In the absence of a client’s Internet connection, the bacula-enterprise-client, libs/common, and qemu plugin packages can be downloaded and manually installed using the dpkg (Debian family) or rpm (RedHat family) commands.
Plugin Configuration
The plugin is configured using Plugin Parameters defined in the Include section of a FileSet in the Bacula Enterprise Director configuration.
Generic Plugin Parameters
The following QEMU Plugin parameters affect any type of Job (Backup, Estimate, or Restore).
- abort_on_error[=<0|1>]
specifies whether the plugin should abort execution (and fail the Bacula Job) if a fatal error occurs during a Backup, Estimate, or Restore operation. This parameter is optional. The default value is 0. - working=</path/to/dir>
specifies the directory used for any QEMU Plugin backup or restore operation. The default value is the WorkingDir configuration parameter of the Bacula FD (check Working Directory Location). This parameter is optional.
Generic QEMU Authentic Estimate and Backup Parameters
These plugin parameters are relevant only for Backup and Estimate jobs:
- qmpcontrol=</path/to/qmp/socket>
specifies the location of the QMP control socket used for backup operations. When this parameter is set, it implies mode=QEMU. This parameter is mandatory for Genuine QEMU and is automatically configured for other virtualization platforms. For other virtualization platforms, this parameter is prohibited. - mode=[QEMU|PVE]
specifies the default QEMU support type. It should be set to QEMU for Genuine QEMU virtualization and PVE for Proxmox virtualization. When not set, QEMU operation mode is configured, which requires a suitable qmpcontrol=… parameter to be configured. - timeout=<seconds>
specifies the time the QEMU Plugin will wait for an image dump to finish. When this timeout is reached, the job will fail. If not set, a default value of 3600 seconds will be used. This parameter is optional. - bitmap-persistence=[0|1]
specifies whether QEMU should persist a dirty block map used for incremental backups on the device itself. Bitmap persistence is supported only on QCOW2 image files. Setting this option on devices that do not support it will cause the backup job to fail. Bitmap persistence is only set during the first Full backup job. For Proxmox, this parameter is set to ‘0’. This parameter is optional.
Important
Ephemeral dirty block bitmap (when bitmap-persistence=0) will be recreated after each VM reboot, forcing a full VM backup.
-
-
- vm=<name>
specifies the name of a guest VM for backup. All guest VMs with the provided <name> will be selected for backup. Multiple vm=… parameters are allowed. If a guest VM with the <name> cannot be found, a single job error will be generated, and the backup will continue to the next VM, unless abort_on_error is set, which will cause the backup job to fail. This parameter is optional and used only when the mode is PVE. - vmid=<vmid>
specifies the VMID of a guest VM for backup. Multiple vmid=… parameters can be provided. If a guest VM with the <vmid> cannot be found, a job error will be generated, and the backup will continue to the next VM, unless abort_on_error is set, which will cause the backup job to fail. This parameter is optional and used only when the mode is PVE. - include=<name-regex>
specifies a list of guest VM names for backup using regular expression syntax. All guest VMs with names that match the provided name-regex expression will be selected for backup. Multiple include=… parameters can be provided. The matching is case-insensitive. If no guest VMs match the provided name-regex expression, the backup will continue to the next parameters or will complete successfully without backing up any VMs. The abort_on_error parameter will not fail the job when no guest VM is found using name matching. This parameter is optional and used only when the mode is PVE. - exclude=<name-regex>
specifies a list of guest VM names that will be excluded from backup using regular expression matching. All guest VMs with names that match the provided expression and were selected for backup using the include=… parameter will be excluded. The matching is case-insensitive. This parameter does not affect any guest VM selected for backup using the vm=…, vmid=…, include=…, or exclude=… parameters. Multiple exclude=… parameters can be provided. This parameter is optional and used only when the mode is PVE.
- vm=<name>
</ul > If none of the vm=…, vmid=…, include=…, and exclude=… parameters are specified, all available guest VMs on the Proxmox hypervisor will be included in the backup. For Genuine QEMU, a single VM pointed to by the qmpcontrol=… parameter will be included in the backup.
-
Working Directory Location
To perform any of the backup or restore job operations, the QEMU Plugin requires some available storage space. The required space depends on the size of the largest disk image file during backup and the sum of the VM disk images during restore. For restore operations, space is needed for proper incremental image patching.
This limitation is a direct consequence of the QEMU QMP backup unit limitation, which supports saved disk image files only as regular files.
The exact location of the workspace can be selected using the working=… plugin parameter. When this parameter is not defined, a default value from the Bacula WorkingDirectory parameter will be used.
This storage space is automatically cleaned up after the operation, unless you select a restore operation without disk image conversion to the destination (check the convdestination restore parameter).
FileSet Examples
As shown in Figure 1, both the FileSet and the QEMU Plugin backup job can be configured through the Bacula Enterprise graphical interface, currently BWeb.
Figure 1. Illustrative example of FileSet for VM backup in BWeb.
In the example below, a single QEMU VM will be backed up.
FileSet {
Name = FS_qemu
Include {
Plugin = "qemu: qmpcontrol=/images/vm1.qmp"
}
}
Now the same QEMU VM, but with a custom working location.
FileSet {
Name = FS_qemu_working
Include {
Plugin = "qemu: qmpcontrol=/images/vm1.qmp working=/tmp"
}
}
In the example below, all Proxmox guest VMs will be backed up.
FileSet {
Name = FS_ProxmoxAll
Include {
Plugin = "qemu: mode=PVE"
}
}
In this example, a single VM with the name “VM1” will be backed up.
FileSet {
Name = FS_Proxmox_VM1
Include {
Plugin = "qemu: mode=PVE vm=VM1"
}
}
The same previous example, but using vmid instead:
FileSet {
Name = FS_Proxmox_VM1
Include {
Plugin = "qemu: mode=PVE vmid=101"
}
}
In the following example, all guest VMs containing “Prod” in their names will be backed up.
FileSet {
Name = FS_Proxmox_ProdAll
Include {
Plugin = "qemu: mode=PVE include=Prod"
}
}
In this final example, all guest VMs except those whose names start with “Test” will be backed up.
FileSet {
Name = FS_Proxmox_AllbutTest
Include {
Plugin = "qemu: mode=PVE include=.* exclude=^Test"
}
}
Plugin Restore Parameters
During restoration, the QEMU Plugin will use the same parameters that were defined for the backup job. These settings are saved in the catalog at the time when the backup job is executed. Some of them can be changed during the restoration process if necessary.
-
-
- convdestination: [yes|no]
specifies whether a restored disk image should be converted to the original format and location as defined during backup. This parameter works only in Genuine QEMU mode. If not defined or set to no (the default), virtual QEMU disk images will be available in the workspace location (see the next parameter) as QCOW2 format files. In this case, you can manually move it to the desired destination or convert it to the desired format. This parameter is optional. - working: </path/to/dir>
specifies the directory used for QEMU Plugin restoration operations, as described in the Working Directory Location section. The default value is the same as for backup and can be set either by the plugin configuration or the Bacula FD WorkingDir configuration parameter. This parameter is optional. - storage: <storage>
specifies a Proxmox Storage to which the restored guest VMs will be restored. If not set, a guest VM will be restored to the Proxmox Storage from which it was backed up. If this parameter points to a non-existent Storage, the original Storage of the guest VM will be used. This parameter is optional.
- convdestination: [yes|no]
-
Practical Restore Examples
As shown in Figure 2, all restoration jobs can be executed through the Bacula Enterprise graphical interface, currently BWeb, from the VM Dashboard or the Restore Menu, or through the CLI commands listed below.
Figure 2. Example of VM Restore through BWeb, “VM Dashboard”.
Restoration with True QEMU
To restore a VM or VMs using the True QEMU mode, you should execute the restore command and specify the “where” parameter as shown in this example:
* restore where=/
Then, proceed to set any other required restoration parameters for the restoration plugin.
In the following example of a restoration session, the restoration option “convdestination” is set to “yes”:
* restore where=/ ... Run Restore job JobName: RestoreFiles Bootstrap: /opt/bacula/working/qemu-test-dir.restore.2.bsr Where: / Replace: Always FileSet: Full Set Backup Client: qemu-test-fd Restore Client: qemu-test-fd Storage: File1 When: 2021-11-01 13:19:16 Catalog: MyCatalog Priority: 10 Plugin Options: *None* OK to run? (yes/mod/no): mod Parameters to modify: 1: Level 2: Storage 3: Job 4: FileSet 5: Restore Client 6: When 7: Priority 8: Bootstrap 9: Where 10: File Relocation 11: Replace 12: JobId 13: Plugin Options Select parameter to modify (1-13): 13 Automatically selected : qemu: qmpcontrol=/images/vm1.qmp abort_on_error Plugin Restore Options Option Current Value Default Value working: *None* (*None*) convdestination: *None* (*None*) Use above plugin configuration? (yes/mod/no): mod You have the following choices: 1: working (Plugin working directory) 2: convdestination (Convert qcow2 restore to original destination) Select parameter to modify (1-2): 2 Please enter a value for convdestination: yes Plugin Restore Options Option Current Value Default Value working: *None* (*None*) convdestination: yes (*None*) Use above plugin configuration? (yes/mod/no): yes ...
During a restore job, you should gather information about each restoration process, including the restored disk devices’ increments and the destination conversion process.
JobId 121: Start Restore Job RestoreFiles.2021-11-01_13.05.11_03 JobId 121: Restoring files from JobId(s) 115,116,120 JobId 121: Using Device "FileChgr1-Dev1" to read. JobId 121: qemu: VM to restore: VM1 (oid:0) JobId 121: qemu: Start Restore vm: VM1 (rid:0) devices: 2 JobId 121: qemu: Restoring device: ide0-hd0 JobId 121: qemu: Restoring device: ide0-hd1 JobId 121: qemu: Patching incremental: ide0-hd0 JobId 121: qemu: Patching incremental: ide0-hd1 JobId 121: qemu: Patching incremental: ide0-hd0 JobId 121: qemu: Patching incremental: ide0-hd1
The new guest VM created during the restoration will be assigned a new VMID (if the original VMID is in use), but the name/hostname will remain the same as it was with the original VM.
Restore to a Proxmox Hypervisor
To restore one or more VMs to a Proxmox hypervisor, you should follow the same restoration command as mentioned above, and the basic restoration process remains the same. The main difference lies in the final configuration and the device import procedure, as shown below:
JobId 117: Start Restore Job RestoreFiles.2021-11-02_17.26.25_07 JobId 117: Restoring files from JobId(s) 104,105,106,115,116 JobId 117: Using Device "FileChgr1-Dev2" to read. JobId 117: qemu: VM to restore: vm1 (oid:100) JobId 117: qemu: Start Restore vm: vm1 (rid:104) devices: 1 JobId 117: qemu: Restoring device: drive-scsi0 JobId 117: qemu: Patching incremental: drive-scsi0 JobId 117: qemu: Patching incremental: drive-scsi0 JobId 117: qemu: Patching incremental: drive-scsi0 JobId 117: qemu: Patching incremental: drive-scsi0 JobId 117: Elapsed time=00:06:42, Transfer rate=20.28 M Bytes/second JobId 117: qemu: Successfully imported device drive-scsi0 as local-lvm:vm-104-disk-0 ...
The new VM created during the restoration will receive a new VMID (if the original VMID is in use), but the name/hostname will remain the same as it was with the original VM.
Restoration to a Local Directory
It’s possible to restore the guest VM’s disk image(s) to a local directory instead of restoring them to a hypervisor as a new VM. To do this, the “where” restoration option should point to a local directory:
* restore where=/tmp/bacula/restores
Take a look at the following example for the “Local VM Restoration” test:
JobId 118: Start Restore Job RestoreFiles.2021-11-02_17.37.34_09 JobId 118: Restoring files from JobId(s) 104,105,106,115,116 JobId 118: Using Device "FileChgr1-Dev1" to read. JobId 118: Ready to read from volume "vol01" on File device "FileChgr1-Dev1" (/opt/bacula/archive). JobId 118: qemu: VM local restore: vm1 (oid:100) JobId 118: Forward spacing Volume "vol01" to addr=227 JobId 118: qemu: Restoring device: drive-scsi0 JobId 118: qemu: Patching incremental: drive-scsi0 JobId 118: qemu: Patching incremental: drive-scsi0 JobId 118: qemu: Patching incremental: drive-scsi0 JobId 118: qemu: Patching incremental: drive-scsi0 JobId 118: Elapsed time=00:06:57, Transfer rate=19.55 M Bytes/second ...
The restoration job log will indicate that the restoration was performed to a local directory.
Others
Resource Listing
The Bacula Enterprise’s QEMU Plugin supports the resource listing feature of Bacula Enterprise 8.x or later. This mode allows a plugin to display useful information about available Proxmox resources, such as:
- List of guest VM names
- List of guest VM VMIDs
- List of Proxmox storages
This feature uses the special “.ls command” with the “plugin=” parameter. The command requires the following parameters to be set:
- client= A Bacula Client name with the QEMU Plugin installed.
- plugin= A plugin name, which would be “qemu:” in this case, with optional plugin parameters as described in the “genericparameters” section.
- path= An object path to be displayed.
The supported values for the “path=” parameter are:
- / to display available object types for listing.
- vm to display a list of guest VM name labels.
- vmid to display a list of guest VM VMIDs and name label pointers.
- storage to show the list of available Storages.
To display available object types, follow the example command below:
*.ls client=proxmoxtest-fd plugin="qemu: mode=PVE" path=/ Connecting to Client proxmoxtest-fd at proxmoxtest:9102 drwxr-x--- 1 root root 0 2021-11-01 12:55:55 vm drwxr-x--- 1 root root 0 2021-11-01 12:55:55 vmid drwxr-x--- 1 root root 0 2021-11-01 12:55:55 storage 2000 OK estimate files=3 bytes=0
To display the list of all available VMs, you can use the following example command:
*.ls client=proxmoxtest-fd plugin="qemu: mode=PVE" path=/vm Connecting to Client proxmoxtest-fd at proxmoxtest:9102 -rw-r----- 1 root root 8589934592 2021-11-01 12:56:18 vm1 -rw-r----- 1 root root 8589934592 2021-11-01 12:56:18 vm3 2000 OK estimate files=2 bytes=17,179,869,184
To display the list of guest VM VMIDs, use the following example command:
*.ls client=proxmoxtest-fd plugin="qemu: mode=PVE" path=/vmid Connecting to Client proxmoxtest-fd at proxmoxtest:9102 -rw-r----- 1 root root 8589934592 2021-11-01 12:56:46 100 -> vm1 -rw-r----- 1 root root 8589934592 2021-11-01 12:56:46 102 -> vm3 2000 OK estimate files=2 bytes=17,179,869,184
The VM and VMID lists display an estimated size of the guest VM based on the sizes of virtual or actual disk images.
To display the available Proxmox Storages, you can use the following example command:
*.ls client=proxmoxtest-fd plugin="qemu: mode=PVE" path=/storage Connecting to Client proxmoxtest-fd at proxmoxtest:9102 brw-r----- 1 root root 0 2021-11-01 12:57:56 data brw-r----- 1 root root 0 2021-11-01 12:57:56 local-lvm 2000 OK estimate files=2 bytes=0
Resource Query
The Bacula Enterprise QEMU Plugin supports the plugin query feature of Bacula Enterprise 14.0 or later. This mode allows a plugin to display the same information about available Proxmox resources that is defined in the resource listing. The Bacula Enterprise QEMU Plugin returns response data in JSON format.
In this example, there is a query about the available VMs:
*.query client=proxmoxtest-fd plugin="qemu: mode=PVE" parameter=vm
[{"name":"vm1","vmid":100},{"name":"vm3","vmid":102}]
*.query client=proxmoxtest-fd plugin="qemu: mode=PVE" parameter=vmid
[{"name":"vm3","vmid":102},{"name":"vm1","vmid":100}]
In this example, there is a query about the available Proxmox storages:
*.query client=proxmoxtest-fd plugin="qemu: mode=PVE" parameter=storage
[{"name":"local"},{"name":"local-lvm"}]
Limitations
- It is not possible to run the same backup job of the same guest VM simultaneously (duplicate job).
- Having different backup jobs that back up the same guest VM is not possible, as this will break the incremental dirty bitmap backup chain. In such a case, successful recovery won’t even be possible.
- Available VM templates in the Proxmox system cannot be saved. This is a limitation of Proxmox.
- In listing and querying mode with the parameters vm or vmid, the plugin will only display running VMs.
- Every VM backup or restoration operation requires sufficient storage space in the working directory (check the Location of the Working Directory). This is a general limitation of the QEMU QMP backup procedure.
References
- QEMU plugin (Bacula System SA)
Disponível em: 
Português (Portuguese (Brazil))
English
Español (Spanish)
