patrick/zfs_autobackup
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fixes

Edwin Eefting
2023-10-15 19:49:06 +02:00
parent 3495dd8986
commit d930be4ad9
1 changed files with 55 additions and 79 deletions
Download Patch File Download Diff File Expand all files Collapse all files

134
Manual.md

@ -24,7 +24,29 @@ zfs-autobackup has safe defaults for use as a backup tool. This means:
* Checking everything, and failing early and verbose.
* Dont do anything unexpected.
Keeping this in mind can help make more sense of the options described here: Most of the options are to modify this safe default behaviour.
Keeping this in mind can help make more sense of the options described here: Most of the options are to modify this safe default behavior.
## Testing and debugging
Its recommended to always use `--verbose` to see whats going on. It makes debugging easier.
During initial setup and testing of a backup you should use `--test`. This will do all the read-only operations, but will not change anything. It will show you exactly what its going to do.
If you encounter a problem and want to see the exact ZFS commands, use `--debug`. This outputs all the underlying zfs-commands in a different color. To even see the output of each command, use `--debug-output`.
Note that debug mode will abort on the first failed dataset, and show a stacktrace. So its not recommended in production.
## Source and target SSH
zfs-autobackup can either work locally are remote via ssh.
The following optional options are for this:
* `--ssh-source USER@HOST`: Source host to pull backup from.
* `--ssh-target USER@HOST`: Target host to push backup to.
If you dont specify a source or target host, zfs-autobackup will operate locally.
Things like different ssh-ports should be configured in your ~/.ssh/config file. (Or the one specified with `--ssh-config CONFIG-FILE`)
## Step 1: Selecting
@ -34,20 +56,20 @@ This step selects the datasets that are part of this run.
Selection is only done by looking at a specific dataset-property. The name of this property is the `backup-name`, formatted by `--property-format`. By default its `autobackup:backup-name`.
The property can have these values:
The actual zfs-property can have the following values:
* `true`: Select the dataset and all its children.
* `false`: Exclude the dataset and all its children.
* `child`: Only select the children of the dataset, not the dataset itself.
* `parent`: Only select the , but not the children. (supported in version 3.2 or higher)
If there are no datasets selected by this property, then zfs-autobackup exists with an error.
If there is no datasets that has this property set, then zfs-autobackup exists with an error.
### Further exclusions
Datasets can also be excluded from selection by these options:
* `--exclude-received`: Exclude datasets that have the origin of their autobackup: property as "received". This can avoid recursive replication between two backup partners.
* `--exclude-received`: Exclude datasets that have the origin of their `autobackup:backup-name` property as "received". This can avoid recursive replication between two backup partners.
* `--exclude-unchanged BYTES`: Exclude datasets that have less than BYTES data changed since any last snapshot. (Use with proxmox HA replication)
## Step 2: Snapshotting
@ -60,15 +82,15 @@ Snapshotting can be skipped with `--no-snapshot`
### Snapshot format
The snapshots are created with a certain naming-format. This includes a timestamp that zfs-autobackup uses to determine when a snapshot can be destroyed by the [Thinner](Thinner.md).
The snapshots are created with a certain naming-format. This includes a timestamp that zfs-autobackup uses to determine when a snapshot can be destroyed by the [Thinner](Thinner).
It is possible to change this format by using `--snapshot-format`. Other snapshots that do not match this format are normally ignored by zfs-autobackup.
It is possible to change this format by using `--snapshot-format`. Other snapshots that do not match this format are normally ignored by zfs-autobackup. Use `--utc` to use UTC for timestamps.
### Pre- and post snapshot commands
You can run commands pre- and post-snapshotting with `--pre-snapshot-cmd` and `--post-snapshot-cmd.`
More info [here](PrePost.md).
More info [here](PrePost).
### Skipping conditions
@ -87,109 +109,63 @@ Syncronisation is done only if `TARGET-PATH` is specified. Otherwise zfs-autobac
For each selected source dataset it does the following steps:
### Planning
### Step 3.1: Planning
If the target dataset already exists:
* We determine the [Common snapshot](Common-snapshots-and-holds.md)
* We determine the [Common snapshot](Common-snapshots-and-holds)
* We check the GUID of the common snaphot, unless `--no-guid-check` is set.
* We determine a list of incompatible snapshots that are in the way. (after our common snapshot)
* If there isn't a valid common snapshot, this dataset fails and we continue with the next one.
We determine which snapshots are kept and which ones can be destroyed by the [Thinner](Thinner.md) once the sync is complete. Note that only our own snapshots (according to --snapshot-format), are considered for deletion.
We determine which snapshots are kept and which ones can be destroyed by the [Thinner](Thinner) once the sync is complete. Note that only our own snapshots (according to --snapshot-format), are considered for deletion.
If `--no-thinning` is used, this list of obsolete snapshots will always be empty.
### Pre-clean
### Step 3.2: Pre-clean
With the plan in hand, we destroy snapshots that are no longer needed on source or target right away, to safe space during sync. (Unless `--no-thinning`)
### Destroy incompatible snapshots
### Step 3.3: Destroy incompatible snapshots
If the planner has detected incompatible snapshots, we will destroy them. But since this can be dangerous and is normally not needed, you have to enable this with `--destroy-incompatible`
Otherwise the dataset will fail.
### Transferring snapshots
### Step 3.4: Transferring snapshots
Now the snapshots are actually transferred, unless `--no-send` is used.
If `--other-snapshots` is specified, we will also transfer snapshots that do not match our `--snapshot-format`. (We will never destroy those)
If `--other-snapshots` is specified, we will also transfer snapshots that do not match our `--snapshot-format`. These other snapshots will never be destroyed.
For each snapshot we:
* Check if we need to resume and aborted transfer.
* Handle [Encryption](Encryption.md) options. (`--encrypt` and `--decrypt`)
* Transfer the data, using various [Transfer options](Piping.md) (`--zfs-compressed`, `--compress`, `--send-pipe`, `--recv-pipe`, `--buffer`, `--rate`)
* Handle [Encryption](Encryption) options. (`--encrypt` and `--decrypt`)
* Transfer the data, using various [Transfer options](Piping) (`--zfs-compressed`, `--compress`, `--send-pipe`, `--recv-pipe`, `--buffer`, `--rate`)
* Filter/set properties according to `--set-properties` and `--filter-properties`
* Add/remove [holds](Common-snapshots-and-holds.md), unless `--no-holds` is used.
* Add/remove [holds](Common-snapshots-and-holds), unless `--no-holds` is used. (Use `--hold-format` to specify the name of this hold)
Just before the first snapshot is transfer, we do a rollback, if `--rollback` is specified.
Just before the first snapshot is transferred, we do a rollback, if `--rollback` is specified.
If its an initial transfer that created a new target dataset, we try to [automount](Mounting.md) the target after the first snapshot is transferred.
If its an initial transfer that created a new target dataset, we try to [automount](Mounting) the target after the first snapshot is transferred.
We destroy obsolete snapshots from the planning phase as soon as possible. (`--no-thinning` effectively disables this )
#### Extra transfer options
* `--ignore-transfer-errors`: ignore zfs transfer errors. It still checks if received filesystem exists. This is usefull to ignore some acltype errors.
* `--clear-refreservation`: Filter "refreservation" property. Recommended, saves space. Same as --filter-properties refreservation.
* `--clear-mountpoint`: Set property canmount=noauto for new datasets. Recommended, prevents mount conflicts. Same as --set-properties canmount=noauto. Also see [Mounting](Mounting)
* `--strip-path N`: Number of directories to strip from target path. An example is giving in the Getting started guide.
* `--force`: Use zfs -F option to force overwrite/rollback. Useful with --strip-path=1, but use with care.
## Step 4: Handle missing datasets
The final step is described [here](Thinner#destroying-missing-datasets) (`--destroy-missing` )
Datasets that are missing or deselected, but still exist in the target-path are called missing datasets.
The handling of those is described [here](Thinner#destroying-missing-datasets) (`--destroy-missing` )
## Thinner
(This page is still work in progress, come back later)
```
################# WIP TODO:
Common options:
--help, -h show help
--test, --dry-run, -n
Dry run, dont change anything, just show what would be done (still does all read-only operations)
--verbose, -v verbose output
--debug, -d Show zfs commands that are executed, stops after an exception.
--debug-output Show zfs commands and their output/exit codes. (noisy)
--progress show zfs progress output. Enabled automaticly on ttys. (use --no-progress to disable)
--utc Use UTC instead of local time when dealing with timestamps for both formatting and parsing. To snapshot in an ISO 8601 compliant time format you may for example specify --snapshot-format "{}-%Y-%m-%dT%H:%M:%SZ". Changing this parameter after-the-fact (existing snapshots) will cause their timestamps to be interpreted as a different time than before.
--version Show version.
SSH options:
--ssh-config CONFIG-FILE
Custom ssh client config
--ssh-source USER@HOST
Source host to pull backup from.
--ssh-target USER@HOST
Target host to push backup to.
String formatting options:
--hold-format FORMAT ZFS hold string format. Default: zfs_autobackup:{}
--strip-path N Number of directories to strip from target path.
Transfer options:
--clear-refreservation
Filter "refreservation" property. (recommended, saves space. same as --filter-properties refreservation)
--clear-mountpoint Set property canmount=noauto for new datasets. (recommended, prevents mount conflicts. same as --set-properties canmount=noauto)
--force, -F Use zfs -F option to force overwrite/rollback. (Useful with --strip-path=1, but use with care)
--ignore-transfer-errors
Ignore transfer errors (still checks if received filesystem exists. useful for acltype errors)
--decrypt Decrypt data before sending it over.
--encrypt Encrypt data after receiving it.
Thinner options:
--no-thinning Do not destroy any snapshots.
--keep-source SCHEDULE
Thinning schedule for old source snapshots. Default: 10,1d1w,1w1m,1m1y
--keep-target SCHEDULE
Thinning schedule for old target snapshots. Default: 10,1d1w,1w1m,1m1y
--destroy-missing SCHEDULE
Destroy datasets on target that are missing on the source. Specify the time since the last snapshot, e.g: --destroy-missing 30d
Full manual at: <https://github.com/psy0rz/zfs_autobackup>
psy@ws2:~/zfs_autobackup$
``````
The thinner decides when a snapshot is obsolete. Look at [Thinner](Thinner) for more info. (`--keep-source` and `--keep-target`)