diff --git a/Manual.md b/Manual.md new file mode 100644 index 0000000..2dc50d7 --- /dev/null +++ b/Manual.md @@ -0,0 +1,190 @@ +## Reference manual + + +zfs-autobackup has multiple steps it goes through. This manual will describe each step and their relevant options. + +### Usage + +``` +zfs-autobackup (options) BACKUP-NAME [TARGET-PATH] +``` + +The only required parameter is `BACKUP-NAME`. + + +## Safe defaults + +zfs-autobackup has safe defaults for use as a backup tool. This means: + +* Preserving all the dataset properties. (which sometimes gets in the way, see [Mounting](Mounting.md)) +* Preserving the full dataset paths. +* Only do stuff with snapshots that match our own format. (ignore the rest) +* Not rolling back or forcing anything. +* Checking everything, and failing early and verbose. +* Dont do anything unexpected. + +Keeping this in mind can help make more sense of the options described here: Most of the options are to modify this safe default behaviour. + +## Step 1: Selecting + +This step selects the datasets that are part of this run. + +### Dataset property + +Selection is only done by looking at a specific dataset-property. The name of this property is the `backup-name`, formatted by `--property-format`. By default its `autobackup:backup-name`. + +The property can have these values: + +* `true`: Select the dataset and all its children. +* `false`: Exclude the dataset and all its children. +* `child`: Only select the children of the dataset, not the dataset itself. +* `parent`: Only select the , but not the children. (supported in version 3.2 or higher) + +If there are no datasets selected by this property, then zfs-autobackup exists with an error. + +### Further exclusions + +Datasets can also be excluded from selection by these options: + +* `--exclude-received`: Exclude datasets that have the origin of their autobackup: property as "received". This can avoid recursive replication between two backup partners. +* `--exclude-unchanged BYTES`: Exclude datasets that have less than BYTES data changed since any last snapshot. (Use with proxmox HA replication) + +## Step 2: Snapshotting + +In this step a snapshot is created on the datasets from step 1. + +zfs-autobackup creates atomic snapshots per pool. e.g. it does one single `zfs snapshot ...` command that includes all the snaphots that need to be taken in a pool. + +Snapshotting can be skipped with `--no-snapshot` + +### Snapshot format + +The snapshots are created with a certain format. This includes a timestamp that zfs-autobackup uses to determine when a snapshot can be destroyed by the [Thinner](Thinner.md). + +It is possible to change this format by using `--snapshot-format`. Other snapshots that do not match this format are normally ignored by zfs-autobackup. + +### Pre- and post snapshot commands + +You can run commands pre- and post-snapshotting with `--pre-snapshot-cmd` and `--post-snapshot-cmd.` + +More info [here](PrePost.md). + + +### Skipping conditions + +Snapshot creation for can be skipped for a dataset, under certain conditions: +* `--min-change BYTES`: Only create snapshot if enough bytes are changed. (default 1) +* `--allow-empty`: If nothing has changed, still create empty snapshots. (Same as --min-change=0) + +### Other options + +* `--set-snapshot-properties PROPERTY=VALUE,...`: List of properties to set on the new snapshot. + +## Step 3: Synchronising + +Syncronisation is done only if `TARGET-PATH` is specified. Otherwise zfs-autobackup is just a snapshot tool and stops after step 2. + +For each selected source dataset it does the following steps: + +### Planning + +If the target dataset already exists: +* We determine the [Common snapshot](Common-snapshots-and-holds.md) +* We check the GUID of the common snaphot, unless `--no-guid-check` is set. +* We determine a list of incompatible snapshots that are in the way. (after our common snapshot) +* If there isn't a valid common snapshot, this dataset fails and we continue with the next one. + +We determine which snapshots are kept and which ones can be destroyed by the [Thinner](Thinner.md) once the sync is complete. Note that only our own snapshots (according to --snapshot-format), are considered for deletion. + +If `--no-thinning` is used, this list of obsolete snapshots will always be empty. + +### Pre-clean + +Which the plan in hand, we destroy snapshots that are no longer needed on source or target right away, to safe space during sync. (Unless `--no-thinning`) + +### Destroy incompatible snapshots + +If the planner has detected incompatible snapshots, we will destroy them. But since this can be dangerous and is normally not needed, you have to enable this with `--destroy-incompatible` + +Otherwise the dataset will fail. + +### Transferring snapshots + +Now the snapshots are actually transferred, unless `--no-send` is used. + +If `--other-snapshots` is specified, we will also transfer snapshots that do not match our `--snapshot-format`. (We will never destroy those) + +For each snapshot we: +* Check if we need to resume and aborted transfer. +* Handle [Encryption](Encryption.md) options. (`--encrypt` and `--decrypt`) +* Transfer the data, using various [Transfer options](Piping.md) (`--zfs-compressed`, `--compress`, `--send-pipe`, `--recv-pipe`, `--buffer`, `--rate`) +* Filter/set properties according to `--set-properties` and `--filter-properties` +* Add/remove [holds](Common-snapshots-and-holds.md), unless `--no-holds` is used. + +Just before the first snapshot is transfer, we do a rollback, if `--rollback` is specified. + +If its an initial transfer that created a new target dataset, we try to [automount](Mounting.md) the target after the first snapshot is transferred. + +We destroy obsolete snapshots from the planning phase as soon as possible. (`--no-thinning` effectively disables this ) + +## Step 4: Handle missing datasets + +The final step is described [here](Thinner#destroying-missing-datasets) (`--destroy-missing` ) + + + +################# + + + +Common options: + --help, -h show help + --test, --dry-run, -n + Dry run, dont change anything, just show what would be done (still does all read-only operations) + --verbose, -v verbose output + --debug, -d Show zfs commands that are executed, stops after an exception. + --debug-output Show zfs commands and their output/exit codes. (noisy) + --progress show zfs progress output. Enabled automaticly on ttys. (use --no-progress to disable) + --utc Use UTC instead of local time when dealing with timestamps for both formatting and parsing. To snapshot in an ISO 8601 compliant time format you may for example specify --snapshot-format "{}-%Y-%m-%dT%H:%M:%SZ". Changing this parameter after-the-fact (existing snapshots) will cause their timestamps to be interpreted as a different time than before. + --version Show version. + +SSH options: + --ssh-config CONFIG-FILE + Custom ssh client config + --ssh-source USER@HOST + Source host to pull backup from. + --ssh-target USER@HOST + Target host to push backup to. + +String formatting options: + + + --hold-format FORMAT ZFS hold string format. Default: zfs_autobackup:{} + --strip-path N Number of directories to strip from target path. + + + + +Transfer options: + --clear-refreservation + Filter "refreservation" property. (recommended, saves space. same as --filter-properties refreservation) + --clear-mountpoint Set property canmount=noauto for new datasets. (recommended, prevents mount conflicts. same as --set-properties canmount=noauto) + --force, -F Use zfs -F option to force overwrite/rollback. (Useful with --strip-path=1, but use with care) + --ignore-transfer-errors + Ignore transfer errors (still checks if received filesystem exists. useful for acltype errors) + --decrypt Decrypt data before sending it over. + --encrypt Encrypt data after receiving it. + + + +Thinner options: + --no-thinning Do not destroy any snapshots. + --keep-source SCHEDULE + Thinning schedule for old source snapshots. Default: 10,1d1w,1w1m,1m1y + --keep-target SCHEDULE + Thinning schedule for old target snapshots. Default: 10,1d1w,1w1m,1m1y + --destroy-missing SCHEDULE + Destroy datasets on target that are missing on the source. Specify the time since the last snapshot, e.g: --destroy-missing 30d + +Full manual at: +psy@ws2:~/zfs_autobackup$ diff --git a/Thinner.md b/Thinner.md index c39bf0e..0933ab2 100644 --- a/Thinner.md +++ b/Thinner.md @@ -63,7 +63,7 @@ If you want to keep ALL the snapshots, just specify a high number. ## Destroying missing datasets -When a dataset has been destroyed or deselected on the source, but still exists on the target we call it a missing dataset. Missing datasets will be still thinned out according to the schedule. +When a dataset has been destroyed or deselected on the source, but still exists on the target we call it a missing dataset. Missing datasets will be still thinned out according to the schedule. (Unless `--no-thinning` is used) The final snapshot will never be destroyed, unless you specify a **deadline** with the `--destroy-missing` option: