update

Common problems.md

@@ -0,0 +1,29 @@
+# Common problems
+
+These are some common ZFS problems you might encounter.
+
+> It keeps asking for my SSH password
+
+You forgot to setup automatic login via SSH keys, look in the example how to do this.
+
+> cannot receive incremental stream: invalid backup stream
+
+This usually means you've created a new snapshot on the target side during a backup. If you restart zfs-autobackup, it will automaticly abort the invalid partially received snapshot and start over.
+
+> cannot receive incremental stream: destination has been modified since most recent snapshot
+
+This means files have been modified on the target side somehow. 
+
+You can use --rollback to automaticly rollback such changes. Also try destroying the target dataset and using --clear-mountpoint on the next run. This way it wont get mounted.
+
+> internal error: Invalid argument
+
+In some cases (Linux -> FreeBSD) this means certain properties are not fully supported on the target system.
+
+Try using something like: --filter-properties xattr or --ignore-transfer-errors. 
+
+>  zfs receive fails, but snapshot seems to be received successful.
+
+This happens if you transfer between different Operating systems/zfs versions or feature sets.
+
+Try using the --ignore-transfer-errors option. This will ignore the error. It will still check if the snapshot is actually received correctly.

Home.md

@@ -230,68 +230,6 @@ root@ws1:~# zfs-autobackup test --verbose
 This also allows you to make several snapshots during the day, but only backup the data at night when the server is not busy.
 
 **Note**: In this mode it doesnt take a specified target-schedule into account when thinning, it only knows a snapshot is the common snapshot by looking at the holds. So make sure your source-schedule keeps the snapshots you still want to transfer at a later point.
-
-
-## More information
-
-* [Thinning out obsolete snapshots](Thinner.md)
-* [Handling ZFS encryption](Encryption.md)
-* [Transfer bufferings, compression and rate limiting.](Pipes.md)
-* [Custom Pre- and post-snapshot commands](Pre+and+post+snapshot+commands.md)
-
-
-
-## Tips
-
-* Use ```--debug``` if something goes wrong and you want to see the commands that are executed. This will also stop at the first error.
-* You can split up the snapshotting and sending tasks by creating two cronjobs. Create a separate snapshotter-cronjob by just omitting target-path.
-* Set the ```readonly``` property of the target filesystem to ```on```. This prevents changes on the target side. (Normally, if there are changes the next backup will fail and will require a zfs rollback.) Note that readonly means you cant change the CONTENTS of the dataset directly. Its still possible to receive new datasets and manipulate properties etc.
-* Use ```--clear-refreservation``` to save space on your backup server.
-* Use ```--clear-mountpoint``` to prevent the target server from mounting the backupped filesystem in the wrong place during a reboot.
-
-### Performance tips
-
-If you have a large number of datasets its important to keep the following tips in mind.
-
-Also it might help to use the --buffer option to add IO buffering during the data transfer. This might speed up things since it smooths out sudden IO bursts that are frequent during a zfs send or recv.
-
-#### Some statistics
-
-To get some idea of how fast zfs-autobackup is, I did some test on my laptop, with a SKHynix_HFS512GD9TNI-L2B0B disk. I'm using zfs 2.0.2.  
-
-I created 100 empty datasets and measured the total runtime of zfs-autobackup. I used all the performance tips below. (--no-holds, --allow-empty, ssh ControlMaster)
-
-* without ssh: 15 seconds. (>6 datasets/s)
-* either ssh-target or ssh-source=localhost: 20 seconds (5 datasets/s)
-* both ssh-target and ssh-source=localhost: 24 seconds (4 datasets/s)
-
-To be bold I created 2500 datasets, but that also was no problem. So it seems it should be possible to use zfs-autobackup with thousands of datasets.
-
-If you need more performance let me know.
-
-NOTE: There is actually a performance regression in ZFS version 2: https://github.com/openzfs/zfs/issues/11560 Use --no-progress as workaround.
-
-#### Less work
-
-You can make zfs-autobackup generate less work by using --no-holds and --allow-empty.
-
-This saves a lot of extra zfs-commands per dataset.
-
-#### Speeding up SSH
-
-You can make your ssh connections persistent and greatly speed up zfs-autobackup:
-
-On the backup-server add this to your ~/.ssh/config:
-
-```console
-Host *
-    ControlPath ~/.ssh/control-master-%r@%h:%p
-    ControlMaster auto
-    ControlPersist 3600
-```
-
-Thanks @mariusvw :)
-
 ### Specifying ssh port or options
 
 The correct way to do this is by creating ~/.ssh/config:
@@ -310,6 +248,23 @@ Also uses compression on slow links.
 
 Look in man ssh_config for many more options.
 
+## Tips
+
+* Use ```--debug``` if something goes wrong and you want to see the commands that are executed. This will also stop at the first error.
+* You can split up the snapshotting and sending tasks by creating two cronjobs. Create a separate snapshotter-cronjob by just omitting target-path.
+* Set the ```readonly``` property of the target filesystem to ```on```. This prevents changes on the target side. (Normally, if there are changes the next backup will fail and will require a zfs rollback.) Note that readonly means you cant change the CONTENTS of the dataset directly. Its still possible to receive new datasets and manipulate properties etc.
+* Use ```--clear-refreservation``` to save space on your backup server.
+* Use ```--clear-mountpoint``` to prevent the target server from mounting the backupped filesystem in the wrong place during a reboot.
+## More information
+
+* [Performance tips (recommended)](Performance%20tips.md)
+* [Common problems and errors](Common%20problems.md)
+* [Thinning out obsolete snapshots](Thinner.md)
+* [Handling ZFS encryption](Encryption.md)
+* [Transfer buffering, compression and rate limiting.](Transfer%20buffering,%20compression%20and%20rate%20limiting.md)
+* [Custom Pre- and post-snapshot commands](Pre%20and%20post%20snapshot%20commands.md)
+
+
 ## Usage
 
 ```console
@@ -433,33 +388,6 @@ Full manual at: https://github.com/psy0rz/zfs_autobackup
 
 ```
 
-## Troubleshooting
-
-### It keeps asking for my SSH password
-
-You forgot to setup automatic login via SSH keys, look in the example how to do this.
-
-### It says 'cannot receive incremental stream: invalid backup stream'
-
-This usually means you've created a new snapshot on the target side during a backup. If you restart zfs-autobackup, it will automaticly abort the invalid partially received snapshot and start over.
-
-### It says 'cannot receive incremental stream: destination has been modified since most recent snapshot'
-
-This means files have been modified on the target side somehow. 
-
-You can use --rollback to automaticly rollback such changes. Also try destroying the target dataset and using --clear-mountpoint on the next run. This way it wont get mounted.
-
-### It says 'internal error: Invalid argument'
-
-In some cases (Linux -> FreeBSD) this means certain properties are not fully supported on the target system.
-
-Try using something like: --filter-properties xattr or --ignore-transfer-errors. 
-
-### zfs receive fails, but snapshot seems to be received successful.
-
-This happens if you transfer between different Operating systems/zfs versions or feature sets.
-
-Try using the --ignore-transfer-errors option. This will ignore the error. It will still check if the snapshot is actually received correctly.
 
 ## Restore example
 

Performance tips.md

@@ -0,0 +1,47 @@
+# Performance tips
+
+If you have a large number of datasets its important to keep the following tips in mind.
+
+## Speeding up SSH
+
+You can make your ssh connections persistent and greatly speed up zfs-autobackup:
+
+On the server that initiates the backup add this to your ~/.ssh/config:
+
+```console
+Host *
+    ControlPath ~/.ssh/control-master-%r@%h:%p
+    ControlMaster auto
+    ControlPersist 3600
+```
+
+Thanks @mariusvw :)
+
+## Buffering
+
+Also it might help to use the --buffer option to use IO buffering during the data transfer. 
+
+This might speed up things since it smooths out sudden IO bursts that are frequent during a zfs send or recv.
+
+## Less work
+
+zfs-autobackup generate a lot less work by using --no-holds and --allow-empty.
+
+This saves a lot of extra zfs-commands per dataset.
+
+## Some statistics
+
+To get some idea of how fast zfs-autobackup is, I did some test on my laptop, with a SKHynix_HFS512GD9TNI-L2B0B disk. I'm using zfs 2.0.2.  
+
+I created 100 empty datasets and measured the total runtime of zfs-autobackup. I used all the performance tips below. (--no-holds, --allow-empty, ssh ControlMaster)
+
+* without ssh: 15 seconds. (>6 datasets/s)
+* either ssh-target or ssh-source=localhost: 20 seconds (5 datasets/s)
+* both ssh-target and ssh-source=localhost: 24 seconds (4 datasets/s)
+
+To be bold I created 2500 datasets, but that also was no problem. So it seems it should be possible to use zfs-autobackup with thousands of datasets.
+
+If you need more performance let me know.
+
+NOTE: There is actually a performance regression in ZFS version 2: https://github.com/openzfs/zfs/issues/11560 Use --no-progress as workaround.
+

Pre and post snapshot commands.md

@@ -2,6 +2,8 @@
 
 You can run commands before and after the snapshot to freeze databases for example, to make the on-disk data consistent before snapshotting.
 
+Note: a ZFS snapshot is atomic, so most of the time freezing isnt really needed. If you restore a snapshot, the application will just think the server (or application) had a regular crash. Most modern databases can handle this fine.
+
 ## Method 1: Use snapshot mode
 
 Its possible to use zfs-autobackup in snapshot-only mode. This way you can just create a script that contains the pre and post steps:
@@ -26,7 +28,7 @@ This has the disadvantage that you might have to do the error handling yourself.
 
 ## Method 2: Use --pre-snapshot-cmd and --post-snapshot-cmd
 
-The commands will be executed on the source side. Use the `--pre-snapshot-cmd` and `--post-snapshot-cmd` options for this.
+With this method, zfs-autobackup will handle the pre and post execution for you.
 
 For example:
 
@@ -37,9 +39,7 @@ zfs-autobackup \
  --ssh-target backupserver backups/db1
 ```
 
-Some rules:
+The way this works:
 * The pre and post commands are always executed on the source side. (via ssh if needed)
-* If a pre-command fails, zfs-autobackup will exit with an error. (after executing the post-commands)
+* If a pre-command fails, it will immediately execute the post-commands and exit with an error code.
-* All post-commands are always executed. Even if the pre-commands or actual snapshot have failed. This way you can be sure that stuff is always cleanedup and unfreezed.
+* All post-commands are always executed. Even if the pre-commands or actual snapshot have failed. This way you can be sure that stuff is always cleaned up and unfreezed.
-
-