diff --git a/Common problems.md b/Common problems.md new file mode 100644 index 0000000..84a73c5 --- /dev/null +++ b/Common problems.md @@ -0,0 +1,29 @@ +# Common problems + +These are some common ZFS problems you might encounter. + +> It keeps asking for my SSH password + +You forgot to setup automatic login via SSH keys, look in the example how to do this. + +> cannot receive incremental stream: invalid backup stream + +This usually means you've created a new snapshot on the target side during a backup. If you restart zfs-autobackup, it will automaticly abort the invalid partially received snapshot and start over. + +> cannot receive incremental stream: destination has been modified since most recent snapshot + +This means files have been modified on the target side somehow. + +You can use --rollback to automaticly rollback such changes. Also try destroying the target dataset and using --clear-mountpoint on the next run. This way it wont get mounted. + +> internal error: Invalid argument + +In some cases (Linux -> FreeBSD) this means certain properties are not fully supported on the target system. + +Try using something like: --filter-properties xattr or --ignore-transfer-errors. + +> zfs receive fails, but snapshot seems to be received successful. + +This happens if you transfer between different Operating systems/zfs versions or feature sets. + +Try using the --ignore-transfer-errors option. This will ignore the error. It will still check if the snapshot is actually received correctly. diff --git a/Home.md b/Home.md index e422218..ea4bab9 100644 --- a/Home.md +++ b/Home.md @@ -230,68 +230,6 @@ root@ws1:~# zfs-autobackup test --verbose This also allows you to make several snapshots during the day, but only backup the data at night when the server is not busy. **Note**: In this mode it doesnt take a specified target-schedule into account when thinning, it only knows a snapshot is the common snapshot by looking at the holds. So make sure your source-schedule keeps the snapshots you still want to transfer at a later point. - - -## More information - -* [Thinning out obsolete snapshots](Thinner.md) -* [Handling ZFS encryption](Encryption.md) -* [Transfer bufferings, compression and rate limiting.](Pipes.md) -* [Custom Pre- and post-snapshot commands](Pre+and+post+snapshot+commands.md) - - - -## Tips - -* Use ```--debug``` if something goes wrong and you want to see the commands that are executed. This will also stop at the first error. -* You can split up the snapshotting and sending tasks by creating two cronjobs. Create a separate snapshotter-cronjob by just omitting target-path. -* Set the ```readonly``` property of the target filesystem to ```on```. This prevents changes on the target side. (Normally, if there are changes the next backup will fail and will require a zfs rollback.) Note that readonly means you cant change the CONTENTS of the dataset directly. Its still possible to receive new datasets and manipulate properties etc. -* Use ```--clear-refreservation``` to save space on your backup server. -* Use ```--clear-mountpoint``` to prevent the target server from mounting the backupped filesystem in the wrong place during a reboot. - -### Performance tips - -If you have a large number of datasets its important to keep the following tips in mind. - -Also it might help to use the --buffer option to add IO buffering during the data transfer. This might speed up things since it smooths out sudden IO bursts that are frequent during a zfs send or recv. - -#### Some statistics - -To get some idea of how fast zfs-autobackup is, I did some test on my laptop, with a SKHynix_HFS512GD9TNI-L2B0B disk. I'm using zfs 2.0.2. - -I created 100 empty datasets and measured the total runtime of zfs-autobackup. I used all the performance tips below. (--no-holds, --allow-empty, ssh ControlMaster) - -* without ssh: 15 seconds. (>6 datasets/s) -* either ssh-target or ssh-source=localhost: 20 seconds (5 datasets/s) -* both ssh-target and ssh-source=localhost: 24 seconds (4 datasets/s) - -To be bold I created 2500 datasets, but that also was no problem. So it seems it should be possible to use zfs-autobackup with thousands of datasets. - -If you need more performance let me know. - -NOTE: There is actually a performance regression in ZFS version 2: https://github.com/openzfs/zfs/issues/11560 Use --no-progress as workaround. - -#### Less work - -You can make zfs-autobackup generate less work by using --no-holds and --allow-empty. - -This saves a lot of extra zfs-commands per dataset. - -#### Speeding up SSH - -You can make your ssh connections persistent and greatly speed up zfs-autobackup: - -On the backup-server add this to your ~/.ssh/config: - -```console -Host * - ControlPath ~/.ssh/control-master-%r@%h:%p - ControlMaster auto - ControlPersist 3600 -``` - -Thanks @mariusvw :) - ### Specifying ssh port or options The correct way to do this is by creating ~/.ssh/config: @@ -310,6 +248,23 @@ Also uses compression on slow links. Look in man ssh_config for many more options. +## Tips + +* Use ```--debug``` if something goes wrong and you want to see the commands that are executed. This will also stop at the first error. +* You can split up the snapshotting and sending tasks by creating two cronjobs. Create a separate snapshotter-cronjob by just omitting target-path. +* Set the ```readonly``` property of the target filesystem to ```on```. This prevents changes on the target side. (Normally, if there are changes the next backup will fail and will require a zfs rollback.) Note that readonly means you cant change the CONTENTS of the dataset directly. Its still possible to receive new datasets and manipulate properties etc. +* Use ```--clear-refreservation``` to save space on your backup server. +* Use ```--clear-mountpoint``` to prevent the target server from mounting the backupped filesystem in the wrong place during a reboot. +## More information + +* [Performance tips (recommended)](Performance%20tips.md) +* [Common problems and errors](Common%20problems.md) +* [Thinning out obsolete snapshots](Thinner.md) +* [Handling ZFS encryption](Encryption.md) +* [Transfer buffering, compression and rate limiting.](Transfer%20buffering,%20compression%20and%20rate%20limiting.md) +* [Custom Pre- and post-snapshot commands](Pre%20and%20post%20snapshot%20commands.md) + + ## Usage ```console @@ -433,33 +388,6 @@ Full manual at: https://github.com/psy0rz/zfs_autobackup ``` -## Troubleshooting - -### It keeps asking for my SSH password - -You forgot to setup automatic login via SSH keys, look in the example how to do this. - -### It says 'cannot receive incremental stream: invalid backup stream' - -This usually means you've created a new snapshot on the target side during a backup. If you restart zfs-autobackup, it will automaticly abort the invalid partially received snapshot and start over. - -### It says 'cannot receive incremental stream: destination has been modified since most recent snapshot' - -This means files have been modified on the target side somehow. - -You can use --rollback to automaticly rollback such changes. Also try destroying the target dataset and using --clear-mountpoint on the next run. This way it wont get mounted. - -### It says 'internal error: Invalid argument' - -In some cases (Linux -> FreeBSD) this means certain properties are not fully supported on the target system. - -Try using something like: --filter-properties xattr or --ignore-transfer-errors. - -### zfs receive fails, but snapshot seems to be received successful. - -This happens if you transfer between different Operating systems/zfs versions or feature sets. - -Try using the --ignore-transfer-errors option. This will ignore the error. It will still check if the snapshot is actually received correctly. ## Restore example diff --git a/Performance tips.md b/Performance tips.md new file mode 100644 index 0000000..75285c9 --- /dev/null +++ b/Performance tips.md @@ -0,0 +1,47 @@ +# Performance tips + +If you have a large number of datasets its important to keep the following tips in mind. + +## Speeding up SSH + +You can make your ssh connections persistent and greatly speed up zfs-autobackup: + +On the server that initiates the backup add this to your ~/.ssh/config: + +```console +Host * + ControlPath ~/.ssh/control-master-%r@%h:%p + ControlMaster auto + ControlPersist 3600 +``` + +Thanks @mariusvw :) + +## Buffering + +Also it might help to use the --buffer option to use IO buffering during the data transfer. + +This might speed up things since it smooths out sudden IO bursts that are frequent during a zfs send or recv. + +## Less work + +zfs-autobackup generate a lot less work by using --no-holds and --allow-empty. + +This saves a lot of extra zfs-commands per dataset. + +## Some statistics + +To get some idea of how fast zfs-autobackup is, I did some test on my laptop, with a SKHynix_HFS512GD9TNI-L2B0B disk. I'm using zfs 2.0.2. + +I created 100 empty datasets and measured the total runtime of zfs-autobackup. I used all the performance tips below. (--no-holds, --allow-empty, ssh ControlMaster) + +* without ssh: 15 seconds. (>6 datasets/s) +* either ssh-target or ssh-source=localhost: 20 seconds (5 datasets/s) +* both ssh-target and ssh-source=localhost: 24 seconds (4 datasets/s) + +To be bold I created 2500 datasets, but that also was no problem. So it seems it should be possible to use zfs-autobackup with thousands of datasets. + +If you need more performance let me know. + +NOTE: There is actually a performance regression in ZFS version 2: https://github.com/openzfs/zfs/issues/11560 Use --no-progress as workaround. + diff --git a/Pre and post snapshot commands.md b/Pre and post snapshot commands.md index 2f22fce..ca57e9e 100644 --- a/Pre and post snapshot commands.md +++ b/Pre and post snapshot commands.md @@ -2,6 +2,8 @@ You can run commands before and after the snapshot to freeze databases for example, to make the on-disk data consistent before snapshotting. +Note: a ZFS snapshot is atomic, so most of the time freezing isnt really needed. If you restore a snapshot, the application will just think the server (or application) had a regular crash. Most modern databases can handle this fine. + ## Method 1: Use snapshot mode Its possible to use zfs-autobackup in snapshot-only mode. This way you can just create a script that contains the pre and post steps: @@ -26,7 +28,7 @@ This has the disadvantage that you might have to do the error handling yourself. ## Method 2: Use --pre-snapshot-cmd and --post-snapshot-cmd -The commands will be executed on the source side. Use the `--pre-snapshot-cmd` and `--post-snapshot-cmd` options for this. +With this method, zfs-autobackup will handle the pre and post execution for you. For example: @@ -37,9 +39,7 @@ zfs-autobackup \ --ssh-target backupserver backups/db1 ``` -Some rules: +The way this works: * The pre and post commands are always executed on the source side. (via ssh if needed) -* If a pre-command fails, zfs-autobackup will exit with an error. (after executing the post-commands) -* All post-commands are always executed. Even if the pre-commands or actual snapshot have failed. This way you can be sure that stuff is always cleanedup and unfreezed. - - +* If a pre-command fails, it will immediately execute the post-commands and exit with an error code. +* All post-commands are always executed. Even if the pre-commands or actual snapshot have failed. This way you can be sure that stuff is always cleaned up and unfreezed. diff --git a/Pipes.md b/Transfer buffering, compression and rate limiting.md similarity index 100% rename from Pipes.md rename to Transfer buffering, compression and rate limiting.md