working on manual

Manual.md

@@ -0,0 +1,190 @@
+## Reference manual
+
+
+zfs-autobackup has multiple steps it goes through. This manual will describe each step and their relevant options.
+
+### Usage
+
+```
+zfs-autobackup (options) BACKUP-NAME [TARGET-PATH]
+```
+
+The only required parameter is `BACKUP-NAME`.
+
+
+## Safe defaults
+
+zfs-autobackup has safe defaults for use as a backup tool. This means:
+
+* Preserving all the dataset properties. (which sometimes gets in the way, see [Mounting](Mounting.md))
+* Preserving the full dataset paths. 
+* Only do stuff with snapshots that match our own format. (ignore the rest)
+* Not rolling back or forcing anything.
+* Checking everything, and failing early and verbose.
+* Dont do anything unexpected.
+
+Keeping this in mind can help make more sense of the options described here: Most of the options are to modify this safe default behaviour.
+
+## Step 1: Selecting
+
+This step selects the datasets that are part of this run.
+
+### Dataset property
+
+Selection is only done by looking at a specific dataset-property. The name of this property is the `backup-name`, formatted by `--property-format`. By default its `autobackup:backup-name`.
+
+The  property can have these values:
+
+* `true`: Select the dataset and all its children.
+* `false`: Exclude the dataset and all its children. 
+* `child`: Only select the children of the dataset, not the dataset itself.
+* `parent`: Only select the , but not the children. (supported in version 3.2 or higher)
+
+If there are no datasets selected by this property, then zfs-autobackup exists with an error.
+
+### Further exclusions
+
+Datasets can also be excluded from selection by these options:
+
+* `--exclude-received`: Exclude datasets that have the origin of their autobackup: property as "received". This can avoid recursive replication between two backup partners.
+* `--exclude-unchanged BYTES`: Exclude datasets that have less than BYTES data changed since any last snapshot. (Use with proxmox HA replication)
+
+## Step 2: Snapshotting
+
+In this step a snapshot is created on the datasets from step 1.
+
+zfs-autobackup creates atomic snapshots per pool. e.g. it does one single `zfs snapshot ...`  command that includes all the snaphots that need to be taken in a pool.
+
+Snapshotting can be skipped with `--no-snapshot`
+
+### Snapshot format
+
+The snapshots are created with a certain format. This includes a timestamp that zfs-autobackup uses to determine when a snapshot can be destroyed by the [Thinner](Thinner.md).
+
+It is possible to change this format by using `--snapshot-format`. Other snapshots that do not match this format are normally ignored by zfs-autobackup.
+
+### Pre- and post snapshot commands
+
+You can run commands pre- and post-snapshotting with `--pre-snapshot-cmd` and `--post-snapshot-cmd.`
+
+More info [here](PrePost.md).
+
+
+### Skipping conditions
+
+Snapshot creation for  can be skipped for a dataset, under certain conditions:
+* `--min-change BYTES`: Only create snapshot if enough bytes are changed. (default 1)
+* `--allow-empty`: If nothing has changed, still create empty snapshots. (Same as --min-change=0)
+
+### Other options
+
+* `--set-snapshot-properties PROPERTY=VALUE,...`: List of properties to set on the new snapshot.
+
+## Step 3: Synchronising
+
+Syncronisation is done only if `TARGET-PATH` is specified. Otherwise zfs-autobackup is just a snapshot tool and stops after step 2.
+
+For each selected source dataset it does the following steps:
+
+### Planning
+
+If the target dataset already exists:
+* We determine the [Common snapshot](Common-snapshots-and-holds.md)
+* We check the GUID of the common snaphot, unless `--no-guid-check` is set.
+* We determine a list of incompatible snapshots that are in the way. (after our common snapshot)
+* If there isn't a valid common snapshot, this dataset fails and we continue with the next one.
+
+We determine which snapshots are kept and which ones can be destroyed by the [Thinner](Thinner.md) once the sync is complete. Note that only our own snapshots (according to --snapshot-format), are considered for deletion.
+
+If `--no-thinning` is used, this list of obsolete snapshots will always be empty.
+
+### Pre-clean
+
+Which the plan in hand, we destroy snapshots that are no longer needed on source or target right away, to safe space during sync. (Unless `--no-thinning`)
+
+### Destroy incompatible snapshots
+
+If the planner has detected incompatible snapshots, we will destroy them. But since this can be dangerous and is normally not needed, you have to enable this with `--destroy-incompatible`
+
+Otherwise the dataset will fail.
+
+### Transferring snapshots
+
+Now the snapshots are actually transferred, unless `--no-send` is used.
+
+If `--other-snapshots` is specified, we will also transfer snapshots that do not match our `--snapshot-format`. (We will never destroy those)
+
+For each snapshot we:
+* Check if we need to resume and aborted transfer.
+* Handle [Encryption](Encryption.md) options. (`--encrypt` and `--decrypt`)
+* Transfer the data, using various [Transfer options](Piping.md) (`--zfs-compressed`, `--compress`, `--send-pipe`, `--recv-pipe`, `--buffer`, `--rate`)
+* Filter/set properties according to `--set-properties` and `--filter-properties`
+* Add/remove [holds](Common-snapshots-and-holds.md), unless `--no-holds` is used.
+
+Just before the first snapshot is transfer, we do a rollback, if `--rollback` is specified.
+
+If its an initial transfer that created a new target dataset, we try to [automount](Mounting.md) the target after the first snapshot is transferred.
+
+We destroy obsolete snapshots from the planning phase as soon as possible. (`--no-thinning` effectively disables this )
+
+## Step 4: Handle missing datasets
+
+The final step is described [here](Thinner#destroying-missing-datasets) (`--destroy-missing` )
+
+
+
+#################
+
+  
+
+Common options:
+  --help, -h            show help
+  --test, --dry-run, -n
+                        Dry run, dont change anything, just show what would be done (still does all read-only operations)
+  --verbose, -v         verbose output
+  --debug, -d           Show zfs commands that are executed, stops after an exception.
+  --debug-output        Show zfs commands and their output/exit codes. (noisy)
+  --progress            show zfs progress output. Enabled automaticly on ttys. (use --no-progress to disable)
+  --utc                 Use UTC instead of local time when dealing with timestamps for both formatting and parsing. To snapshot in an ISO 8601 compliant time format you may for example specify   --snapshot-format "{}-%Y-%m-%dT%H:%M:%SZ". Changing this parameter after-the-fact (existing snapshots) will cause their timestamps to be interpreted as a different time than before.
+  --version             Show version.
+
+SSH options:
+  --ssh-config CONFIG-FILE
+                        Custom ssh client config
+  --ssh-source USER@HOST
+                        Source host to pull backup from.
+  --ssh-target USER@HOST
+                        Target host to push backup to.
+
+String formatting options:
+
+
+  --hold-format FORMAT  ZFS hold string format. Default: zfs_autobackup:{}
+  --strip-path N        Number of directories to strip from target path.
+
+
+
+
+Transfer options:
+  --clear-refreservation
+                        Filter "refreservation" property. (recommended, saves space. same as --filter-properties refreservation)
+  --clear-mountpoint    Set property canmount=noauto for new datasets. (recommended, prevents mount conflicts. same as --set-properties canmount=noauto)
+  --force, -F           Use zfs -F option to force overwrite/rollback. (Useful with --strip-path=1, but use with care)
+  --ignore-transfer-errors
+                        Ignore transfer errors (still checks if received filesystem exists. useful for acltype errors)
+  --decrypt             Decrypt data before sending it over.
+  --encrypt             Encrypt data after receiving it.
+
+
+
+Thinner options:
+  --no-thinning         Do not destroy any snapshots.
+  --keep-source SCHEDULE
+                        Thinning schedule for old source snapshots. Default: 10,1d1w,1w1m,1m1y
+  --keep-target SCHEDULE
+                        Thinning schedule for old target snapshots. Default: 10,1d1w,1w1m,1m1y
+  --destroy-missing SCHEDULE
+                        Destroy datasets on target that are missing on the source. Specify the time since the last snapshot, e.g: --destroy-missing 30d
+
+Full manual at: <https://github.com/psy0rz/zfs_autobackup>
+psy@ws2:~/zfs_autobackup$

Thinner.md

@@ -63,7 +63,7 @@ If you want to keep ALL the snapshots, just specify a high number.
 
 ## Destroying missing datasets
 
-When a dataset has been destroyed or deselected on the source, but still exists on the target we call it a missing dataset. Missing datasets will be still thinned out according to the schedule.
+When a dataset has been destroyed or deselected on the source, but still exists on the target we call it a missing dataset. Missing datasets will be still thinned out according to the schedule. (Unless `--no-thinning` is used)
 
 The final snapshot will never be destroyed, unless you specify a **deadline** with the `--destroy-missing` option: