From 41122e2b157163450063a3fa8858749ce0cdce7f Mon Sep 17 00:00:00 2001 From: Dan MacDonald Date: Sun, 15 Oct 2023 12:08:34 +0100 Subject: [PATCH 01/15] Updated Home (markdown) --- Home.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Home.md b/Home.md index bae5a91..c36b633 100644 --- a/Home.md +++ b/Home.md @@ -6,7 +6,7 @@ Look at the [README.md](../blob/master/README.md) for the introduction. zfs-autobackup creates ZFS snapshots on a "source" machine and then replicates those snapshots to a "target" machine via SSH. -zfs-autobackup may be installed on either the source machine or the target machine. (Installing on both is unnecessary.) +zfs-autobackup may be installed on either the source machine or the target machine, installing it on both is unnecessary. When installed on the source, zfs-autobackup will push snapshots to the target. When installed on the target, zfs-autobackup will pull snapshots from the source. From 6d31c788088d7d83b1dba8ed0eca730c9f0f80b1 Mon Sep 17 00:00:00 2001 From: Dan MacDonald Date: Sun, 15 Oct 2023 12:10:12 +0100 Subject: [PATCH 02/15] Updated Home (markdown) --- Home.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Home.md b/Home.md index c36b633..16d07d3 100644 --- a/Home.md +++ b/Home.md @@ -32,7 +32,7 @@ On older machines you might have to use easy_install: ### Using the sources -If you dont want to install zfs-autobackup, or want to make some changes to the code, look at [Development](Development) +If you don't want to install zfs-autobackup, or want to make some changes to the code, look at [Development](Development) ## Example From e82799c410a009b9050372cf7623833da12fdd85 Mon Sep 17 00:00:00 2001 From: Dan MacDonald Date: Sun, 15 Oct 2023 12:17:34 +0100 Subject: [PATCH 03/15] Typos --- Home.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Home.md b/Home.md index 16d07d3..594b559 100644 --- a/Home.md +++ b/Home.md @@ -82,7 +82,7 @@ This allows the backup machine to login to `pve01` as root without password. Next, we specify the filesystems we want to snapshot and replicate by assigning a unique group name to those filesystems. -Its important to choose a unique group name and to use the name consistently. (Advanced tip: If you have multiple sets of filesystems that you wish to backup differently, you may do this by creating multiple group names.) +It's important to choose a unique group name and to use the name consistently. (Advanced tip: If you have multiple sets of filesystems that you wish to backup differently, you may do this by creating multiple group names.) In this example, we assign the group name `offsite1` to the filesystems we want to backup. @@ -183,15 +183,15 @@ Run the script on the backup machine and pull the data from the source machine s ### The results -As you might notice, zfs-autobackup preserve the whole parent-path of the source. +As you might notice, zfs-autobackup preserves the whole parent path of the source. So `rpool/data/vm100-disk-0` ends up as: `data/backup/pve01/rpool/data/vm-100-disk-0` -Since its a backup, its usefull to preserve the original structure of the data like this. +Since it's a backup, it's useful to preserve the original structure of the data like this. ### Stripping the path -Since you might think this is ugly, there is the `--strip-path` option. However this can lead to collisions if you 2 source datasets result in the same target paths. Since version 3.1.2 zfs-autobackup will check for this and emit an error. +Since you might think this is ugly, there is the `--strip-path` option. However this can lead to collisions if two source datasets result in the same target paths. Since version 3.1.2 zfs-autobackup will check for this and emit an error. #### Making source and target paths look the same From 32f8bff001f81c42b03caef0f83c1b4d5802be1d Mon Sep 17 00:00:00 2001 From: Dan MacDonald Date: Sun, 15 Oct 2023 12:22:07 +0100 Subject: [PATCH 04/15] Updated Home (markdown) --- Home.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/Home.md b/Home.md index 594b559..8f5d7a2 100644 --- a/Home.md +++ b/Home.md @@ -217,9 +217,9 @@ Note that you can always change the ssh source and target parameters at a later #### Pull+push (zero trust) -It also possible to use a 3rd server that pulls backups from the source and pushes the data to the target server via 1 stream. This way the source and target server wont have to be able to reach each other. If one server gets hacked, they cant access the other server. +It also possible to use a third server that pulls backups from the source and pushes the data to the target server via one stream. This way the source and target server won't have to be able to reach each other. If one server gets hacked, they can't access the other server. -To do this, you only have to install zfs-autobackup on a 3rd server and use both `--ssh-source` and `--ssh-target` to specify the other source and target servers. +To do this, you only have to install zfs-autobackup on a third server and use both `--ssh-source` and `--ssh-target` to specify the other source and target servers. ## Local Usage @@ -239,9 +239,7 @@ Now every time you run the command, zfs-autobackup will create a new snapshot an Older snapshots will eventually be deleted, depending on the `--keep-source` and `--keep-target` settings. The defaults are shown above under the 'Settings summary'. Look at [Thinner](Thinner) for more info. -Once you've got the correct settings for your situation, you can just store the command in a cronjob. - -Or just create a script and run it manually when you need it. +Once you've got the correct settings for your situation, you can just store the command in a cronjob or just create a script and run it manually when you need it. ## Monitoring From 7585b0bd489065c812ed84a3b70d975c4ca7ea0b Mon Sep 17 00:00:00 2001 From: Dan MacDonald Date: Sun, 15 Oct 2023 12:27:37 +0100 Subject: [PATCH 05/15] Updated Home (markdown) --- Home.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Home.md b/Home.md index 8f5d7a2..ee2e660 100644 --- a/Home.md +++ b/Home.md @@ -245,7 +245,7 @@ Once you've got the correct settings for your situation, you can just store the Don't forget to monitor the results of your backups, look at [Monitoring](Monitoring) for more info. -## Splitting up snapshot and backup job +## Splitting up snapshot and backup jobs You might want to make snapshots during the week, and only transfer data during the weekends. @@ -259,7 +259,7 @@ And this on weekend days: You can also create the snapshots in offline mode by using zfs-autobackup as a snapshot tool on the source side. This way the snapshots will always be created, even if the backup server is offline or unreachable. -## Use as snapshot tool +## Use as a snapshot tool You can use zfs-autobackup as a standalone snapshot tool. @@ -296,7 +296,7 @@ You CAN use the same backup name to transfer data to multiple targets. However i One way to do this is to make adjust the --keep-source option or to make sure the backups run at a close enough interval. -However: To prevent confusion, and to be more flexible, I would advise to always use different and clear to distinguish names. e.g.: autobackup:offsite and autobackup:local for example. +To prevent confusion, and to be more flexible, I would advise you to always use easily distinguished names e.g.: autobackup:offsite and autobackup:local, for example. ## Tips From 1330ccba42d8e099b2d3e56208b4f0d2eae3fe20 Mon Sep 17 00:00:00 2001 From: Dan MacDonald Date: Sun, 15 Oct 2023 12:30:02 +0100 Subject: [PATCH 06/15] Updated Home (markdown) --- Home.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Home.md b/Home.md index ee2e660..1066692 100644 --- a/Home.md +++ b/Home.md @@ -302,8 +302,8 @@ To prevent confusion, and to be more flexible, I would advise you to always use * Use ```--clear-mountpoint``` to prevent all kinds of problems. See [[Mounting]] * Use ```--debug``` if something goes wrong and you want to see the commands that are executed. This will also stop at the first error. -* Use these only one time if needed: `--force` `--destroy-incompatible` `--rollback`. Dont add them to your script. Try to solve the underlying cause if you keep needing them. -* Set the ```readonly``` property of the target filesystem to ```on```. This prevents changes on the target side. (Due to the nature of ZFS itself, if any changes are made to a dataset on the target machine, then the next backup to that target machine will probably fail. Such a failure can probably be resolved by perfroming a target-side zfs rollback of the affected dataset.) Note that ```readonly``` prevents changes to the CONTENTS of the dataset directly. Its still possible to receive new datasets and manipulate properties etc. +* Use these only one time if needed: `--force` `--destroy-incompatible` `--rollback`. Don't add them to your script. Try to solve the underlying cause if you keep needing them. +* Set the ```readonly``` property of the target filesystem to ```on```. This prevents changes on the target side. (Due to the nature of ZFS itself, if any changes are made to a dataset on the target machine, then the next backup to that target machine will probably fail. Such a failure can probably be resolved by perfroming a target-side zfs rollback of the affected dataset.) Note that ```readonly``` prevents changes to the CONTENTS of the dataset directly. It's still possible to receive new datasets and manipulate properties etc. * Use ```--clear-refreservation``` to save space on your backup machine. * zfs-autobackup uses holds by default, so you might get "dataset busy" if you try to destroy a snapshot. (check zfs holds --help) From 8aa8ff5562c8e849511246c8fd1a363d49cb3498 Mon Sep 17 00:00:00 2001 From: Dan MacDonald Date: Sun, 15 Oct 2023 12:34:36 +0100 Subject: [PATCH 07/15] Add link to "dataset is busy" wiki page --- Home.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Home.md b/Home.md index 1066692..fee15dd 100644 --- a/Home.md +++ b/Home.md @@ -305,7 +305,7 @@ To prevent confusion, and to be more flexible, I would advise you to always use * Use these only one time if needed: `--force` `--destroy-incompatible` `--rollback`. Don't add them to your script. Try to solve the underlying cause if you keep needing them. * Set the ```readonly``` property of the target filesystem to ```on```. This prevents changes on the target side. (Due to the nature of ZFS itself, if any changes are made to a dataset on the target machine, then the next backup to that target machine will probably fail. Such a failure can probably be resolved by perfroming a target-side zfs rollback of the affected dataset.) Note that ```readonly``` prevents changes to the CONTENTS of the dataset directly. It's still possible to receive new datasets and manipulate properties etc. * Use ```--clear-refreservation``` to save space on your backup machine. -* zfs-autobackup uses holds by default, so you might get "dataset busy" if you try to destroy a snapshot. (check zfs holds --help) +* zfs-autobackup uses holds by default, so you might get "dataset busy" if you try to destroy a snapshot. (check zfs holds --help or see [https://github.com/psy0rz/zfs_autobackup/wiki/Problems#dataset-is-busy](here.) ## Restore example From db00976cd9ca31267d3e54f56edf8cf03c0e1bec Mon Sep 17 00:00:00 2001 From: Dan MacDonald Date: Sun, 15 Oct 2023 12:37:45 +0100 Subject: [PATCH 08/15] Updated Home (markdown) --- Home.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Home.md b/Home.md index fee15dd..ab94914 100644 --- a/Home.md +++ b/Home.md @@ -305,7 +305,7 @@ To prevent confusion, and to be more flexible, I would advise you to always use * Use these only one time if needed: `--force` `--destroy-incompatible` `--rollback`. Don't add them to your script. Try to solve the underlying cause if you keep needing them. * Set the ```readonly``` property of the target filesystem to ```on```. This prevents changes on the target side. (Due to the nature of ZFS itself, if any changes are made to a dataset on the target machine, then the next backup to that target machine will probably fail. Such a failure can probably be resolved by perfroming a target-side zfs rollback of the affected dataset.) Note that ```readonly``` prevents changes to the CONTENTS of the dataset directly. It's still possible to receive new datasets and manipulate properties etc. * Use ```--clear-refreservation``` to save space on your backup machine. -* zfs-autobackup uses holds by default, so you might get "dataset busy" if you try to destroy a snapshot. (check zfs holds --help or see [https://github.com/psy0rz/zfs_autobackup/wiki/Problems#dataset-is-busy](here.) +* zfs-autobackup uses holds by default, so you might get "dataset busy" if you try to destroy a snapshot. (check zfs holds --help or see [https://github.com/psy0rz/zfs_autobackup/wiki/Problems#dataset-is-busy](here.)) ## Restore example From d0fea9ddd38f0c612b9ab1dfd35c959e4b56d2b9 Mon Sep 17 00:00:00 2001 From: Dan MacDonald Date: Sun, 15 Oct 2023 12:41:12 +0100 Subject: [PATCH 09/15] Fix link --- Home.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Home.md b/Home.md index ab94914..8f9140b 100644 --- a/Home.md +++ b/Home.md @@ -305,7 +305,7 @@ To prevent confusion, and to be more flexible, I would advise you to always use * Use these only one time if needed: `--force` `--destroy-incompatible` `--rollback`. Don't add them to your script. Try to solve the underlying cause if you keep needing them. * Set the ```readonly``` property of the target filesystem to ```on```. This prevents changes on the target side. (Due to the nature of ZFS itself, if any changes are made to a dataset on the target machine, then the next backup to that target machine will probably fail. Such a failure can probably be resolved by perfroming a target-side zfs rollback of the affected dataset.) Note that ```readonly``` prevents changes to the CONTENTS of the dataset directly. It's still possible to receive new datasets and manipulate properties etc. * Use ```--clear-refreservation``` to save space on your backup machine. -* zfs-autobackup uses holds by default, so you might get "dataset busy" if you try to destroy a snapshot. (check zfs holds --help or see [https://github.com/psy0rz/zfs_autobackup/wiki/Problems#dataset-is-busy](here.)) +* zfs-autobackup uses holds by default, so you might get "dataset busy" if you try to destroy a snapshot. (check zfs holds --help or see [here.](https://github.com/psy0rz/zfs_autobackup/wiki/Problems#dataset-is-busy)) ## Restore example From 86e49961d63baf1735e79c1fe3cc4d36cf2bca4a Mon Sep 17 00:00:00 2001 From: Dan MacDonald Date: Sun, 15 Oct 2023 12:47:18 +0100 Subject: [PATCH 10/15] With v3.2, I did not need to use --force on my first sync when using --strip-path even though I still had an existing (but empty) dataset so I made it clear that --force is optional and to use --test first --- Home.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Home.md b/Home.md index 8f9140b..3adae7f 100644 --- a/Home.md +++ b/Home.md @@ -200,7 +200,7 @@ If you want your source and target structure to look exactly the same, you have * Select the whole source-pool. In this case: `zfs set autobackup:offsite1=true rpool` * Use `--strip-path=1` * Specify target-pool as target-path. In this case: `data` -* Use the `--force` option the first time to overwrite the existing target pool. (New in v3.1.2) +* You may need to use `--force` option the first time to overwrite the existing target pool. (New in v3.1.2). It is recommended you try with `--test` and without `--force` first. ### Pull or push? From 06a1afb7e3b085d79c11df456d5b8620459f381d Mon Sep 17 00:00:00 2001 From: Dan MacDonald Date: Sun, 15 Oct 2023 12:49:07 +0100 Subject: [PATCH 11/15] Updated Home (markdown) --- Home.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Home.md b/Home.md index 3adae7f..6020a04 100644 --- a/Home.md +++ b/Home.md @@ -200,7 +200,7 @@ If you want your source and target structure to look exactly the same, you have * Select the whole source-pool. In this case: `zfs set autobackup:offsite1=true rpool` * Use `--strip-path=1` * Specify target-pool as target-path. In this case: `data` -* You may need to use `--force` option the first time to overwrite the existing target pool. (New in v3.1.2). It is recommended you try with `--test` and without `--force` first. +* You may need to use `--force` option the first time to overwrite the existing target pool. It is recommended you try with `--test` and without `--force` first (New in v3.1.2). ### Pull or push? From b1b1993ec1fdd6d6e3c8d9fea8ecbcadce5e2608 Mon Sep 17 00:00:00 2001 From: Dan MacDonald Date: Sun, 15 Oct 2023 12:55:02 +0100 Subject: [PATCH 12/15] Add notice about excluding datasets when using --strip-path --- Home.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/Home.md b/Home.md index 6020a04..da47f80 100644 --- a/Home.md +++ b/Home.md @@ -202,6 +202,14 @@ If you want your source and target structure to look exactly the same, you have * Specify target-pool as target-path. In this case: `data` * You may need to use `--force` option the first time to overwrite the existing target pool. It is recommended you try with `--test` and without `--force` first (New in v3.1.2). +This configuration will attempt replicate the entire pool from the source to the target. If you wish to exclude specific datasets from being replicated from the source pool, make sure that you do so by running commands such as: + +```console +[root@pve01 ~]# zfs set autobackup:offsite1=false rpool/tmp +``` + +For each dataset you don't want to replicate BEFORE you run zfs-autobackup without `--test` for the first time. + ### Pull or push? Note that this is called a "pull" backup. The backup (target) machine pulls the backup from the source machine. This is usually the preferred way. From f8dc9f98fb9da68852ae977a8aec0dc925b70c4c Mon Sep 17 00:00:00 2001 From: Dan MacDonald Date: Sun, 15 Oct 2023 13:11:47 +0100 Subject: [PATCH 13/15] Add "Use alongside other snapshot tools" section --- Home.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/Home.md b/Home.md index da47f80..3e15d26 100644 --- a/Home.md +++ b/Home.md @@ -279,6 +279,10 @@ Only use this if you don't want to make any backup at all, or if a target isn't If you have offline backups, checkout [[Common-snapshots-and-holds]] +## Use alongside other snapshot tools + +zfs-autobackup can happily co-exist on the same system as other ZFS snapshot tools such as zfs-auto-snapshot if you are already using one. zfs-autobackup will not thin any manually created snapshots or those created by other snapshot tools, it will only thin its own shapshots if you use its `--keep-source` or `--keep-target` options. + ## Specifying ssh port or options The correct way to do this is by creating ~/.ssh/config: From b74015455221050f8caed250a449c778ff595574 Mon Sep 17 00:00:00 2001 From: Dan MacDonald Date: Sun, 15 Oct 2023 13:38:34 +0100 Subject: [PATCH 14/15] Add running without root section --- Home.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/Home.md b/Home.md index 3e15d26..86fda83 100644 --- a/Home.md +++ b/Home.md @@ -78,6 +78,22 @@ root@backup:~# ``` This allows the backup machine to login to `pve01` as root without password. +### Running without root + +In order to run zfs-autobackup without root permissions, you'll need to set a few ZFS permissions. The permissions required differ for receiving and sending. + +On the machine you want to sync the dataset from, you'll need the send, hold, mount, snapshot, and destroy permissions. You can apply them like so: + +```console +root@backup:~# zfs allow -u localuser mount,send,hold,snapshot,destroy rpool +``` + +On the receiving side, you will need the compression, mountpoint, create, mount, receive, rollback and destroy permissions: + +```console +root@backup:~# zfs allow -u remoteuser compression,mountpoint,create,mount,receive,rollback,destroy tank/backups/rpool +``` + ### Select filesystems to backup Next, we specify the filesystems we want to snapshot and replicate by assigning a unique group name to those filesystems. From 9357c2475014c20a4327951a7adec7b625a11ce8 Mon Sep 17 00:00:00 2001 From: Dan MacDonald Date: Sun, 15 Oct 2023 14:07:24 +0100 Subject: [PATCH 15/15] Updated Home (markdown) --- Home.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Home.md b/Home.md index 86fda83..3892a29 100644 --- a/Home.md +++ b/Home.md @@ -85,13 +85,13 @@ In order to run zfs-autobackup without root permissions, you'll need to set a fe On the machine you want to sync the dataset from, you'll need the send, hold, mount, snapshot, and destroy permissions. You can apply them like so: ```console -root@backup:~# zfs allow -u localuser mount,send,hold,snapshot,destroy rpool +root@source:~# zfs allow -u localuser mount,send,hold,snapshot,destroy rpool ``` On the receiving side, you will need the compression, mountpoint, create, mount, receive, rollback and destroy permissions: ```console -root@backup:~# zfs allow -u remoteuser compression,mountpoint,create,mount,receive,rollback,destroy tank/backups/rpool +root@target:~# zfs allow -u remoteuser compression,mountpoint,create,mount,receive,rollback,destroy tank/backups/rpool ``` ### Select filesystems to backup