patrick/zfs_autobackup
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

update

Edwin Eefting
2022-01-06 23:14:27 +01:00
parent c73c8484d4
commit dd9313cfbb
5 changed files with 99 additions and 95 deletions
Download Patch File Download Diff File Expand all files Collapse all files

29
Common problems.md Normal file

@ -0,0 +1,29 @@
# Common problems
These are some common ZFS problems you might encounter.
> It keeps asking for my SSH password
You forgot to setup automatic login via SSH keys, look in the example how to do this.
> cannot receive incremental stream: invalid backup stream
This usually means you've created a new snapshot on the target side during a backup. If you restart zfs-autobackup, it will automaticly abort the invalid partially received snapshot and start over.
> cannot receive incremental stream: destination has been modified since most recent snapshot
This means files have been modified on the target side somehow.
You can use --rollback to automaticly rollback such changes. Also try destroying the target dataset and using --clear-mountpoint on the next run. This way it wont get mounted.
> internal error: Invalid argument
In some cases (Linux -> FreeBSD) this means certain properties are not fully supported on the target system.
Try using something like: --filter-properties xattr or --ignore-transfer-errors.
> zfs receive fails, but snapshot seems to be received successful.
This happens if you transfer between different Operating systems/zfs versions or feature sets.
Try using the --ignore-transfer-errors option. This will ignore the error. It will still check if the snapshot is actually received correctly.

106
Home.md

@ -230,68 +230,6 @@ root@ws1:~# zfs-autobackup test --verbose
This also allows you to make several snapshots during the day, but only backup the data at night when the server is not busy.
**Note**: In this mode it doesnt take a specified target-schedule into account when thinning, it only knows a snapshot is the common snapshot by looking at the holds. So make sure your source-schedule keeps the snapshots you still want to transfer at a later point.
## More information
* [Thinning out obsolete snapshots](Thinner.md)
* [Handling ZFS encryption](Encryption.md)
* [Transfer bufferings, compression and rate limiting.](Pipes.md)
* [Custom Pre- and post-snapshot commands](Pre+and+post+snapshot+commands.md)
## Tips
* Use ```--debug``` if something goes wrong and you want to see the commands that are executed. This will also stop at the first error.
* You can split up the snapshotting and sending tasks by creating two cronjobs. Create a separate snapshotter-cronjob by just omitting target-path.
* Set the ```readonly``` property of the target filesystem to ```on```. This prevents changes on the target side. (Normally, if there are changes the next backup will fail and will require a zfs rollback.) Note that readonly means you cant change the CONTENTS of the dataset directly. Its still possible to receive new datasets and manipulate properties etc.
* Use ```--clear-refreservation``` to save space on your backup server.
* Use ```--clear-mountpoint``` to prevent the target server from mounting the backupped filesystem in the wrong place during a reboot.
### Performance tips
If you have a large number of datasets its important to keep the following tips in mind.
Also it might help to use the --buffer option to add IO buffering during the data transfer. This might speed up things since it smooths out sudden IO bursts that are frequent during a zfs send or recv.
#### Some statistics
To get some idea of how fast zfs-autobackup is, I did some test on my laptop, with a SKHynix_HFS512GD9TNI-L2B0B disk. I'm using zfs 2.0.2.
I created 100 empty datasets and measured the total runtime of zfs-autobackup. I used all the performance tips below. (--no-holds, --allow-empty, ssh ControlMaster)
* without ssh: 15 seconds. (>6 datasets/s)
* either ssh-target or ssh-source=localhost: 20 seconds (5 datasets/s)
* both ssh-target and ssh-source=localhost: 24 seconds (4 datasets/s)
To be bold I created 2500 datasets, but that also was no problem. So it seems it should be possible to use zfs-autobackup with thousands of datasets.
If you need more performance let me know.
NOTE: There is actually a performance regression in ZFS version 2: https://github.com/openzfs/zfs/issues/11560 Use --no-progress as workaround.
#### Less work
You can make zfs-autobackup generate less work by using --no-holds and --allow-empty.
This saves a lot of extra zfs-commands per dataset.
#### Speeding up SSH
You can make your ssh connections persistent and greatly speed up zfs-autobackup:
On the backup-server add this to your ~/.ssh/config:
```console
Host *
ControlPath ~/.ssh/control-master-%r@%h:%p
ControlMaster auto
ControlPersist 3600
```
Thanks @mariusvw :)
### Specifying ssh port or options
The correct way to do this is by creating ~/.ssh/config:
@ -310,6 +248,23 @@ Also uses compression on slow links.
Look in man ssh_config for many more options.
## Tips
* Use ```--debug``` if something goes wrong and you want to see the commands that are executed. This will also stop at the first error.
* You can split up the snapshotting and sending tasks by creating two cronjobs. Create a separate snapshotter-cronjob by just omitting target-path.
* Set the ```readonly``` property of the target filesystem to ```on```. This prevents changes on the target side. (Normally, if there are changes the next backup will fail and will require a zfs rollback.) Note that readonly means you cant change the CONTENTS of the dataset directly. Its still possible to receive new datasets and manipulate properties etc.
* Use ```--clear-refreservation``` to save space on your backup server.
* Use ```--clear-mountpoint``` to prevent the target server from mounting the backupped filesystem in the wrong place during a reboot.
## More information
* [Performance tips (recommended)](Performance%20tips.md)
* [Common problems and errors](Common%20problems.md)
* [Thinning out obsolete snapshots](Thinner.md)
* [Handling ZFS encryption](Encryption.md)
* [Transfer buffering, compression and rate limiting.](Transfer%20buffering,%20compression%20and%20rate%20limiting.md)
* [Custom Pre- and post-snapshot commands](Pre%20and%20post%20snapshot%20commands.md)
## Usage
```console
@ -433,33 +388,6 @@ Full manual at: https://github.com/psy0rz/zfs_autobackup
```
## Troubleshooting
### It keeps asking for my SSH password
You forgot to setup automatic login via SSH keys, look in the example how to do this.
### It says 'cannot receive incremental stream: invalid backup stream'
This usually means you've created a new snapshot on the target side during a backup. If you restart zfs-autobackup, it will automaticly abort the invalid partially received snapshot and start over.
### It says 'cannot receive incremental stream: destination has been modified since most recent snapshot'
This means files have been modified on the target side somehow.
You can use --rollback to automaticly rollback such changes. Also try destroying the target dataset and using --clear-mountpoint on the next run. This way it wont get mounted.
### It says 'internal error: Invalid argument'
In some cases (Linux -> FreeBSD) this means certain properties are not fully supported on the target system.
Try using something like: --filter-properties xattr or --ignore-transfer-errors.
### zfs receive fails, but snapshot seems to be received successful.
This happens if you transfer between different Operating systems/zfs versions or feature sets.
Try using the --ignore-transfer-errors option. This will ignore the error. It will still check if the snapshot is actually received correctly.
## Restore example

47
Performance tips.md Normal file

@ -0,0 +1,47 @@
# Performance tips
If you have a large number of datasets its important to keep the following tips in mind.
## Speeding up SSH
You can make your ssh connections persistent and greatly speed up zfs-autobackup:
On the server that initiates the backup add this to your ~/.ssh/config:
```console
Host *
ControlPath ~/.ssh/control-master-%r@%h:%p
ControlMaster auto
ControlPersist 3600
```
Thanks @mariusvw :)
## Buffering
Also it might help to use the --buffer option to use IO buffering during the data transfer.
This might speed up things since it smooths out sudden IO bursts that are frequent during a zfs send or recv.
## Less work
zfs-autobackup generate a lot less work by using --no-holds and --allow-empty.
This saves a lot of extra zfs-commands per dataset.
## Some statistics
To get some idea of how fast zfs-autobackup is, I did some test on my laptop, with a SKHynix_HFS512GD9TNI-L2B0B disk. I'm using zfs 2.0.2.
I created 100 empty datasets and measured the total runtime of zfs-autobackup. I used all the performance tips below. (--no-holds, --allow-empty, ssh ControlMaster)
* without ssh: 15 seconds. (>6 datasets/s)
* either ssh-target or ssh-source=localhost: 20 seconds (5 datasets/s)
* both ssh-target and ssh-source=localhost: 24 seconds (4 datasets/s)
To be bold I created 2500 datasets, but that also was no problem. So it seems it should be possible to use zfs-autobackup with thousands of datasets.
If you need more performance let me know.
NOTE: There is actually a performance regression in ZFS version 2: https://github.com/openzfs/zfs/issues/11560 Use --no-progress as workaround.

12
Pre and post snapshot commands.md

@ -2,6 +2,8 @@
You can run commands before and after the snapshot to freeze databases for example, to make the on-disk data consistent before snapshotting.
Note: a ZFS snapshot is atomic, so most of the time freezing isnt really needed. If you restore a snapshot, the application will just think the server (or application) had a regular crash. Most modern databases can handle this fine.
## Method 1: Use snapshot mode
Its possible to use zfs-autobackup in snapshot-only mode. This way you can just create a script that contains the pre and post steps:
@ -26,7 +28,7 @@ This has the disadvantage that you might have to do the error handling yourself.
## Method 2: Use --pre-snapshot-cmd and --post-snapshot-cmd
The commands will be executed on the source side. Use the `--pre-snapshot-cmd` and `--post-snapshot-cmd` options for this.
With this method, zfs-autobackup will handle the pre and post execution for you.
For example:
@ -37,9 +39,7 @@ zfs-autobackup \
--ssh-target backupserver backups/db1
```
Some rules:
The way this works:
* The pre and post commands are always executed on the source side. (via ssh if needed)
* If a pre-command fails, zfs-autobackup will exit with an error. (after executing the post-commands)
* All post-commands are always executed. Even if the pre-commands or actual snapshot have failed. This way you can be sure that stuff is always cleanedup and unfreezed.
* If a pre-command fails, it will immediately execute the post-commands and exit with an error code.
* All post-commands are always executed. Even if the pre-commands or actual snapshot have failed. This way you can be sure that stuff is always cleaned up and unfreezed.

0
Pipes.md → Transfer buffering, compression and rate limiting.md