fix linter isue

Feature/ssh keygen

README.md

@@ -17,6 +17,7 @@
 * More robust error handling.
 * Prepared for future enhanchements.
 * Supports raw backups for encryption.
+* Custom SSH client config.
 
 ## Introduction
 
@@ -87,13 +88,69 @@ It should work with python 2.7 and higher.
 
 ## Example
 
-In this example we're going to backup a machine called `pve` to our backupserver.
+In this example we're going to backup a machine called `pve` to a machine called `backup`.
 
-Its important to choose a unique and consistent backup name. In this case we name our backup: `offsite1`.
+### Setup SSH login
+
+zfs-autobackup needs passwordless login via ssh. This means generating an ssh key and copying it to the remote server.
+
+#### Generate SSH key on `backup`
+
+On the server that runs zfs-autobackup you need to create an SSH key. You only need to do this once.
+
+Use the `ssh-keygen` command and leave the passphrase empty:
+
+```console
+root@backup:~# ssh-keygen
+Generating public/private rsa key pair.
+Enter file in which to save the key (/root/.ssh/id_rsa):
+Enter passphrase (empty for no passphrase):
+Enter same passphrase again:
+Your identification has been saved in /root/.ssh/id_rsa.
+Your public key has been saved in /root/.ssh/id_rsa.pub.
+The key fingerprint is:
+SHA256:McJhCxvaxvFhO/3e8Lf5gzSrlTWew7/bwrd2U2EHymE root@backup
+The key's randomart image is:
++---[RSA 2048]----+
+|    + =          |
+|   + X *    E .  |
+|  . = B +  o o . |
+|   .   o +  o  o.|
+|        S o   .oo|
+|         . + o= +|
+|          . ++==.|
+|            .+o**|
+|           .. +B@|
++----[SHA256]-----+
+root@backup:~#
+```
+
+#### Copy SSH key to `pve`
+
+Now you need to copy the public part of the key to `pve`
+
+The `ssh-copy-id` command is a handy tool to automate this. It will just ask for your password.
+
+```console
+root@backup:~# ssh-copy-id root@pve.server.com
+/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
+/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
+/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
+Password:
+
+Number of key(s) added: 1
+
+Now try logging into the machine, with:   "ssh 'root@pve.server.com'"
+and check to make sure that only the key(s) you wanted were added.
+
+root@backup:~#
+```
 
 ### Select filesystems to backup
 
-On the source zfs system set the ```autobackup:offsite``` zfs property to true:
+Its important to choose a unique and consistent backup name. In this case we name our backup: `offsite1`.
+
+On the source zfs system set the ```autobackup:offsite1``` zfs property to true:
 
 ```console
 [root@pve ~]# zfs set autobackup:offsite1=true rpool
@@ -125,13 +182,7 @@ rpool/swap                              autobackup:offsite1  false
 
 ### Running zfs-autobackup
 
-Before you start, make sure you can login to the server without password, by using `SSH keys`. Look at the troubleshooting section for more info.
-
-There are 2 ways to run the backup, but the endresult is always the same. Its just a matter of security (trust relations between the servers) and preference.
-
-#### Method 1: Pull backup
-
-Run the script on the backup server and pull the data from the server specfied by --ssh-source. This is usually the preferred way and prevents a hacked server from accesing the backup-data.
+Run the script on the backup server and pull the data from the server specfied by --ssh-source.
 
 ```console
 [root@backup ~]# zfs-autobackup --ssh-source pve.server.com offsite1 backup/pve --progress --verbose
@@ -143,14 +194,14 @@ Run the script on the backup server and pull the data from the server specfied b
   [Source] Keep oldest of 1 week, delete after 1 month.
   [Source] Keep oldest of 1 month, delete after 1 year.
   [Source] Send all datasets that have 'autobackup:offsite1=true' or 'autobackup:offsite1=child'
-  
+
   [Target] Datasets are local
   [Target] Keep the last 10 snapshots.
   [Target] Keep oldest of 1 day, delete after 1 week.
   [Target] Keep oldest of 1 week, delete after 1 month.
   [Target] Keep oldest of 1 month, delete after 1 year.
   [Target] Receive datasets under: backup/pve
-  
+
   #### Selecting
   [Source] rpool: Selected (direct selection)
   [Source] rpool/ROOT: Selected (inherited selection)
@@ -158,14 +209,14 @@ Run the script on the backup server and pull the data from the server specfied b
   [Source] rpool/data: Selected (inherited selection)
   [Source] rpool/data/vm-100-disk-0: Selected (inherited selection)
   [Source] rpool/swap: Ignored (disabled)
-  
+
   #### Snapshotting
   [Source] rpool: No changes since offsite1-20200218175435
   [Source] rpool/ROOT: No changes since offsite1-20200218175435
   [Source] rpool/data: No changes since offsite1-20200218175435
   [Source] Creating snapshot offsite1-20200218180123
-  
-  #### Transferring
+
+  #### Sending and thinning
   [Target] backup/pve/rpool/ROOT/pve-1@offsite1-20200218175435: receiving full
   [Target] backup/pve/rpool/ROOT/pve-1@offsite1-20200218175547: receiving incremental
   [Target] backup/pve/rpool/ROOT/pve-1@offsite1-20200218175706: receiving incremental
@@ -176,30 +227,9 @@ Run the script on the backup server and pull the data from the server specfied b
   ...
 ```
 
-#### Method 2: push backup
+Note that this is called a "pull" backup: The backup server pulls the backup from the server. This is usually the preferred way.
 
-Run the script on the server and push the data to the backup server specified by --ssh-target.
-
-```console
-[root@pve ~]# zfs-autobackup --ssh-target backup.server.com offsite1 backup/pve --progress --verbose
-
-  #### Settings summary
-  [Source] Datasets are local
-  [Source] Keep the last 10 snapshots.
-  [Source] Keep oldest of 1 day, delete after 1 week.
-  [Source] Keep oldest of 1 week, delete after 1 month.
-  [Source] Keep oldest of 1 month, delete after 1 year.
-  [Source] Send all datasets that have 'autobackup:offsite1=true' or 'autobackup:offsite1=child'
-  
-  [Target] Datasets on: backup.server.com
-  [Target] Keep the last 10 snapshots.
-  [Target] Keep oldest of 1 day, delete after 1 week.
-  [Target] Keep oldest of 1 week, delete after 1 month.
-  [Target] Keep oldest of 1 month, delete after 1 year.
-  [Target] Receive datasets under: backup/pve
-  ...
-
-```
+Its also possible to let a server push its backup to the backup-server. However this has security implications. In that case you would setup the SSH keys the other way around and use the --ssh-target parameter on the server.
 
 ### Automatic backups
 
@@ -207,18 +237,51 @@ Now everytime you run the command, zfs-autobackup will create a new snapshot and
 
 Older snapshots will evertually be deleted, depending on the `--keep-source` and `--keep-target` settings. (The defaults are shown above under the 'Settings summary')
 
-Once you've got the correct settings for your situation, you can just store the command in a cronjob. 
+Once you've got the correct settings for your situation, you can just store the command in a cronjob.
 
 Or just create a script and run it manually when you need it.
 
 ## Tips
 
-* Use ```--verbose``` to see details, otherwise zfs-autobackup will be quiet and only show errors, like a nice unix command.
 * Use ```--debug``` if something goes wrong and you want to see the commands that are executed. This will also stop at the first error.
-* Use ```--resume``` to be able to resume aborted backups. (not all zfs versions support this)
+* You can split up the snapshotting and sending tasks by creating two cronjobs. Use ```--no-send``` for the snapshotter-cronjob and use ```--no-snapshot``` for the send-cronjob. This is usefull if you only want to send at night or if your send take too long.
 * Set the ```readonly``` property of the target filesystem to ```on```. This prevents changes on the target side. (Normally, if there are changes the next backup will fail and will require a zfs rollback.) Note that readonly means you cant change the CONTENTS of the dataset directly. Its still possible to receive new datasets and manipulate properties etc.
 * Use ```--clear-refreservation``` to save space on your backup server.
 * Use ```--clear-mountpoint``` to prevent the target server from mounting the backupped filesystem in the wrong place during a reboot.
+* Use ```--resume``` to be able to resume aborted backups. (not all zfs versions support this)
+
+### Speeding up SSH
+
+You can make your ssh connections persistent and greatly speed up zfs-autobackup:
+
+On the backup-server add this to your ~/.ssh/config:
+
+```console
+Host *
+    ControlPath ~/.ssh/control-master-%r@%h:%p
+    ControlMaster auto
+    ControlPersist 3600
+```
+
+Thanks @mariusvw :)
+
+### Specifying ssh port or options
+
+The correct way to do this is by creating ~/.ssh/config:
+
+```console
+Host smartos04
+    Hostname 1.2.3.4
+    Port 1234
+    user root
+    Compression yes
+```
+
+This way you can just specify "smartos04" as host.
+
+Also uses compression on slow links.
+
+Look in man ssh_config for many more options.
 
 ## Usage
 
@@ -226,11 +289,12 @@ Here you find all the options:
 
 ```console
 [root@server ~]# zfs-autobackup --help
-usage: zfs-autobackup [-h] [--ssh-source SSH_SOURCE] [--ssh-target SSH_TARGET]
-                      [--keep-source KEEP_SOURCE] [--keep-target KEEP_TARGET]
-                      [--no-snapshot] [--allow-empty] [--ignore-replicated]
-                      [--no-holds] [--resume] [--strip-path STRIP_PATH]
-                      [--buffer BUFFER] [--clear-refreservation]
+usage: zfs-autobackup [-h] [--ssh-config SSH_CONFIG] [--ssh-source SSH_SOURCE]
+                      [--ssh-target SSH_TARGET] [--keep-source KEEP_SOURCE]
+                      [--keep-target KEEP_TARGET] [--other-snapshots]
+                      [--no-snapshot] [--no-send] [--allow-empty]
+                      [--ignore-replicated] [--no-holds] [--resume]
+                      [--strip-path STRIP_PATH] [--clear-refreservation]
                       [--clear-mountpoint]
                       [--filter-properties FILTER_PROPERTIES]
                       [--set-properties SET_PROPERTIES] [--rollback]
@@ -238,7 +302,7 @@ usage: zfs-autobackup [-h] [--ssh-source SSH_SOURCE] [--ssh-target SSH_TARGET]
                       [--debug] [--debug-output] [--progress]
                       backup_name target_path
 
-ZFS autobackup 3.0-rc3
+zfs-autobackup v3.0-rc6 - Copyright 2020 E.H.Eefting (edwin@datux.nl)
 
 positional arguments:
   backup_name           Name of the backup (you should set the zfs property
@@ -248,6 +312,8 @@ positional arguments:
 
 optional arguments:
   -h, --help            show this help message and exit
+  --ssh-config SSH_CONFIG
+                        Custom ssh client config
   --ssh-source SSH_SOURCE
                         Source host to get backup from. (user@hostname)
                         Default None.
@@ -260,30 +326,33 @@ optional arguments:
   --keep-target KEEP_TARGET
                         Thinning schedule for old target snapshots. Default:
                         10,1d1w,1w1m,1m1y
-  --no-snapshot         dont create new snapshot (usefull for finishing
+  --other-snapshots     Send over other snapshots as well, not just the ones
+                        created by this tool.
+  --no-snapshot         Dont create new snapshots (usefull for finishing
                         uncompleted backups, or cleanups)
-  --allow-empty         if nothing has changed, still create empty snapshots.
+  --no-send             Dont send snapshots (usefull for cleanups, or if you
+                        want a serperate send-cronjob)
+  --allow-empty         If nothing has changed, still create empty snapshots.
   --ignore-replicated   Ignore datasets that seem to be replicated some other
                         way. (No changes since lastest snapshot. Usefull for
                         proxmox HA replication)
   --no-holds            Dont lock snapshots on the source. (Usefull to allow
                         proxmox HA replication to switches nodes)
-  --resume              support resuming of interrupted transfers by using the
+  --resume              Support resuming of interrupted transfers by using the
                         zfs extensible_dataset feature (both zpools should
                         have it enabled) Disadvantage is that you need to use
                         zfs recv -A if another snapshot is created on the
                         target during a receive. Otherwise it will keep
                         failing.
   --strip-path STRIP_PATH
-                        number of directory to strip from path (use 1 when
+                        Number of directory to strip from path (use 1 when
                         cloning zones between 2 SmartOS machines)
   --clear-refreservation
                         Filter "refreservation" property. (recommended, safes
                         space. same as --filter-properties refreservation)
-  --clear-mountpoint    Filter "canmount" property. You still have to set
-                        canmount=noauto on the backup server. (recommended,
-                        prevents mount conflicts. same as --filter-properties
-                        canmount)
+  --clear-mountpoint    Set property canmount=noauto for new datasets.
+                        (recommended, prevents mount conflicts. same as --set-
+                        properties canmount=noauto)
   --filter-properties FILTER_PROPERTIES
                         List of propererties to "filter" when receiving
                         filesystems. (you can still restore them with zfs
@@ -310,61 +379,22 @@ optional arguments:
 
 When a filesystem fails, zfs_backup will continue and report the number of
 failures at that end. Also the exit code will indicate the number of failures.
-
 ```
 
-
-### Speeding up SSH and prevent connection flooding
-
-Add this to your ~/.ssh/config:
-
-```console
-Host *
-    ControlPath ~/.ssh/control-master-%r@%h:%p
-    ControlMaster auto
-    ControlPersist 3600
-```
-
-This will make all your ssh connections persistent and greatly speed up zfs-autobackup for jobs with short intervals.
-
-Thanks @mariusvw :)
-
-### Specifying ssh port or options
-
-The correct way to do this is by creating ~/.ssh/config:
-
-```console
-Host smartos04
-    Hostname 1.2.3.4
-    Port 1234
-    user root
-    Compression yes
-```
-
-This way you can just specify "smartos04" as host.
-
-Also uses compression on slow links.
-
-Look in man ssh_config for many more options.
-
 ## Troubleshooting
 
 ### It keeps asking for my SSH password
 
-You forgot to setup automatic login via SSH keys:
+You forgot to setup automatic login via SSH keys, look in the example how to do this.
 
-* Create a SSH key on the server that you want to run zfs-autobackup on. Use `ssh-keygen`.
-* Copy the public key to your clipboard. Get it with `cat /root/.ssh/id_rsa.pub`
-* Add the key to the server you specified with --ssh-source or --ssh-target. Create and add it to `/root/.ssh/authorized_keys`
-
-> ###  cannot receive incremental stream: invalid backup stream
+### It says 'cannot receive incremental stream: invalid backup stream'
 
 This usually means you've created a new snapshot on the target side during a backup:
 
 * Solution 1: Restart zfs-autobackup and make sure you dont use --resume. If you did use --resume, be sure to "abort" the recveive on the target side with zfs recv -A.
 * Solution 2: Destroy the newly created snapshot and restart zfs-autobackup.
 
-> ### internal error: Invalid argument
+### It says 'internal error: Invalid argument'
 
 In some cases (Linux -> FreeBSD) this means certain properties are not fully supported on the target system.
 

bin/zfs-autobackup

@@ -13,22 +13,21 @@ import re
 import traceback
 import subprocess
 import pprint
-# import cStringIO
 import time
 import argparse
 from pprint import pprint as p
 import select
 
+use_color=False
+if sys.stdout.isatty():
+    try:
+        import colorama
+        use_color=True
+    except ImportError:
+        pass
 
-import imp
-try:
-    import colorama
-    use_color=True
-except ImportError:
-    use_color=False
-
-VERSION="3.0-rc4"
-
+VERSION="3.0-rc7"
+HEADER="zfs-autobackup v{} - Copyright 2020 E.H.Eefting (edwin@datux.nl)\n".format(VERSION)
 
 class Log:
     def __init__(self, show_debug=False, show_verbose=False):
@@ -307,12 +306,14 @@ class ExecuteNode:
     """an endpoint to execute local or remote commands via ssh"""
 
 
-    def __init__(self, ssh_to=None, readonly=False, debug_output=False):
-        """ssh_to: server you want to ssh to. none means local
+    def __init__(self, ssh_config=None, ssh_to=None, readonly=False, debug_output=False):
+        """ssh_config: custom ssh config
+           ssh_to: server you want to ssh to. none means local
            readonly: only execute commands that dont make any changes (usefull for testing-runs)
            debug_output: show output and exit codes of commands in debugging output.
         """
 
+        self.ssh_config=ssh_config
         self.ssh_to=ssh_to
         self.readonly=readonly
         self.debug_output=debug_output
@@ -357,7 +358,12 @@ class ExecuteNode:
 
         #use ssh?
         if self.ssh_to != None:
-            encoded_cmd.extend(["ssh".encode('utf-8'), self.ssh_to.encode('utf-8')])
+            encoded_cmd.append("ssh".encode('utf-8'))
+
+            if self.ssh_config != None:
+                encoded_cmd.extend(["-F".encode('utf-8'), self.ssh_config.encode('utf-8')])
+
+            encoded_cmd.append(self.ssh_to.encode('utf-8'))
 
             #make sure the command gets all the data in utf8 format:
             #(this is neccesary if LC_ALL=en_US.utf8 is not set in the environment)
@@ -492,7 +498,6 @@ class ZfsDataset():
         'volume': [ "canmount" ],
     }
 
-    ZFS_MAX_UNCHANGED_BYTES=200000
 
     def __init__(self, zfs_node, name, force_exists=None):
         """name: full path of the zfs dataset
@@ -509,6 +514,9 @@ class ZfsDataset():
         return(self.name)
 
     def __eq__(self, obj):
+        if not isinstance(obj, ZfsDataset):
+            return(False)
+
         return(self.name == obj.name)
 
     def verbose(self,txt):
@@ -581,36 +589,35 @@ class ZfsDataset():
             return(ZfsDataset(self.zfs_node, self.rstrip_path(1)))
 
 
-    def find_our_prev_snapshot(self, snapshot):
-        """find our previous snapshot in this dataset. None if it doesnt exist"""
+    def find_prev_snapshot(self, snapshot, other_snapshots=False):
+        """find previous snapshot in this dataset. None if it doesnt exist.
+
+        other_snapshots: set to true to also return snapshots that where not created by us. (is_ours)
+        """
 
         if self.is_snapshot:
             raise(Exception("Please call this on a dataset."))
 
-        try:
-            index=self.find_our_snapshot_index(snapshot)
-            if index!=None and index>0:
-                return(self.our_snapshots[index-1])
-            else:
-                return(None)
-        except:
-            return(None)
+        index=self.find_snapshot_index(snapshot)
+        while index:
+            index=index-1
+            if other_snapshots or self.snapshots[index].is_ours():
+                return(self.snapshots[index])
+        return(None)
 
 
-    def find_our_next_snapshot(self, snapshot):
-        """find our next snapshot in this dataset. None if it doesnt exist"""
+    def find_next_snapshot(self, snapshot, other_snapshots=False):
+        """find next snapshot in this dataset. None if it doesnt exist"""
 
         if self.is_snapshot:
             raise(Exception("Please call this on a dataset."))
 
-        try:
-            index=self.find_our_snapshot_index(snapshot)
-            if index!=None and index>=0 and index<len(self.our_snapshots)-1:
-                return(self.our_snapshots[index+1])
-            else:
-                return(None)
-        except:
-            return(None)
+        index=self.find_snapshot_index(snapshot)
+        while index!=None and index<len(self.snapshots)-1:
+            index=index+1
+            if other_snapshots or self.snapshots[index].is_ours():
+                return(self.snapshots[index])
+        return(None)
 
 
     @cached_property
@@ -644,9 +651,13 @@ class ZfsDataset():
 
     def destroy(self, fail_exception=False):
         """destroy the dataset. by default failures are not an exception, so we can continue making backups"""
+
         self.verbose("Destroying")
+
+        self.release()
+
         try:
-            self.zfs_node.run(["zfs", "destroy", self.name])
+            self.zfs_node.run(["zfs", "destroy", "-d", self.name])
             self.invalidate()
             self.force_exists=False
             return(True)
@@ -679,12 +690,14 @@ class ZfsDataset():
         return(ret)
 
 
-    def is_changed(self):
+    def is_changed(self, min_changed_bytes=1):
         """dataset is changed since ANY latest snapshot ?"""
         self.debug("Checking if dataset is changed")
 
-        #NOTE: filesystems can have a very small amount written without actual changes in some cases
-        if int(self.properties['written'])<=self.ZFS_MAX_UNCHANGED_BYTES:
+        if min_changed_bytes==0:
+            return(True)
+
+        if int(self.properties['written'])<min_changed_bytes:
             return(False)
         else:
             return(True)
@@ -698,16 +711,35 @@ class ZfsDataset():
             return(False)
 
 
+    @property
+    def _hold_name(self):
+        return("zfs_autobackup:"+self.zfs_node.backup_name)
+
+
+    @property
+    def holds(self):
+        """get list of holds for dataset"""
+
+        output=self.zfs_node.run([ "zfs" , "holds", "-H", self.name ], valid_exitcodes=[ 0 ], tab_split=True, readonly=True)
+        return(map(lambda fields: fields[1], output))
+
+
+    def is_hold(self):
+        """did we hold this snapshot?"""
+        return(self._hold_name in self.holds)
+
+
     def hold(self):
         """hold dataset"""
         self.debug("holding")
-        self.zfs_node.run([ "zfs" , "hold", "zfs_autobackup:"+self.zfs_node.backup_name, self.name ], valid_exitcodes=[ 0,1 ])
+        self.zfs_node.run([ "zfs" , "hold", self._hold_name, self.name ], valid_exitcodes=[ 0,1 ])
+
 
     def release(self):
         """release dataset"""
-        self.debug("releasing")
-        self.zfs_node.run([ "zfs" , "release", "zfs_autobackup:"+self.zfs_node.backup_name, self.name ], valid_exitcodes=[ 0,1 ])
-
+        if self.zfs_node.readonly or self.is_hold():
+            self.debug("releasing")
+            self.zfs_node.run([ "zfs" , "release", self._hold_name, self.name ], valid_exitcodes=[ 0,1 ])
 
 
     @property
@@ -757,22 +789,22 @@ class ZfsDataset():
 
 
     def find_snapshot(self, snapshot):
-        """find snapshot by snapshot (can be a snapshot_name or ZfsDataset)"""
+        """find snapshot by snapshot (can be a snapshot_name or a different ZfsDataset )"""
 
         if not isinstance(snapshot,ZfsDataset):
             snapshot_name=snapshot
         else:
             snapshot_name=snapshot.snapshot_name
 
-        for snapshot in self.our_snapshots:
+        for snapshot in self.snapshots:
             if snapshot.snapshot_name==snapshot_name:
                 return(snapshot)
 
         return(None)
 
 
-    def find_our_snapshot_index(self, snapshot):
-        """find our snapshot index by snapshot (can be a snapshot_name or ZfsDataset)"""
+    def find_snapshot_index(self, snapshot):
+        """find snapshot index by snapshot (can be a snapshot_name or ZfsDataset)"""
 
         if not isinstance(snapshot,ZfsDataset):
             snapshot_name=snapshot
@@ -780,7 +812,7 @@ class ZfsDataset():
             snapshot_name=snapshot.snapshot_name
 
         index=0
-        for snapshot in self.our_snapshots:
+        for snapshot in self.snapshots:
             if snapshot.snapshot_name==snapshot_name:
                 return(index)
             index=index+1
@@ -789,24 +821,35 @@ class ZfsDataset():
 
 
     @cached_property
-    def is_changed_ours(self):
-        """dataset is changed since OUR latest snapshot?"""
-
-        self.debug("Checking if dataset is changed since our snapshot")
-
-        if not self.our_snapshots:
-            return(True)
+    def written_since_ours(self):
+        """get number of bytes written since our last snapshot"""
+        self.debug("Getting bytes written since our last snapshot")
 
         latest_snapshot=self.our_snapshots[-1]
 
         cmd=[ "zfs", "get","-H" ,"-ovalue", "-p", "written@"+str(latest_snapshot), self.name ]
+
         output=self.zfs_node.run(readonly=True, tab_split=False, cmd=cmd, valid_exitcodes=[ 0 ])
+
+        return(int(output[0]))
+
+
+    def is_changed_ours(self, min_changed_bytes=1):
+        """dataset is changed since OUR latest snapshot?"""
+
+        if min_changed_bytes==0:
+            return(True)
+
+        if not self.our_snapshots:
+            return(True)
+
         #NOTE: filesystems can have a very small amount written without actual changes in some cases
-        if int(output[0])<=self.ZFS_MAX_UNCHANGED_BYTES:
+        if self.written_since_ours<min_changed_bytes:
             return(False)
 
         return(True)
 
+
     @cached_property
     def recursive_datasets(self, types="filesystem,volume"):
         """get all datasets recursively under us"""
@@ -992,25 +1035,24 @@ class ZfsDataset():
         """find latest coommon snapshot between us and target
         returns None if its an initial transfer
         """
-        if not target_dataset.our_snapshots:
+        if not target_dataset.snapshots:
             #target has nothing yet
             return(None)
         else:
-            snapshot=self.find_snapshot(target_dataset.our_snapshots[-1].snapshot_name)
+            # snapshot=self.find_snapshot(target_dataset.snapshots[-1].snapshot_name)
 
-            if not snapshot:
-                #try to find another common snapshot as rollback-suggestion for admin
-                for target_snapshot in reversed(target_dataset.our_snapshots):
-                    if self.find_snapshot(target_snapshot):
-                        target_snapshot.error("Latest common snapshot, roll back to this.")
-                        raise(Exception("Cant find latest target snapshot on source."))
-                target_dataset.error("Cant find common snapshot with target. ")
-                raise(Exception("You probablly need to delete the target dataset to fix this."))
+            # if not snapshot:
+            #try to common snapshot
+            for target_snapshot in reversed(target_dataset.snapshots):
+                if self.find_snapshot(target_snapshot):
+                    target_snapshot.debug("common snapshot")
+                    return(target_snapshot)
+                    # target_snapshot.error("Latest common snapshot, roll back to this.")
+                    # raise(Exception("Cant find latest target snapshot on source."))
+            target_dataset.error("Cant find common snapshot with target. ")
+            raise(Exception("You probablly need to delete the target dataset to fix this."))
 
 
-            snapshot.debug("common snapshot")
-
-            return(snapshot)
 
     def get_allowed_properties(self, filter_properties, set_properties):
         """only returns lists of allowed properties for this dataset type"""
@@ -1030,21 +1072,75 @@ class ZfsDataset():
         return ( ( allowed_filter_properties, allowed_set_properties  )  )
 
 
-    def sync_snapshots(self, target_dataset, show_progress=False, resume=True,  filter_properties=[], set_properties=[], ignore_recv_exit_code=False, source_holds=True, rollback=False, raw=False):
-        """sync this dataset's snapshots to target_dataset,"""
-
+    def sync_snapshots(self, target_dataset, show_progress=False, resume=True,  filter_properties=[], set_properties=[], ignore_recv_exit_code=False, source_holds=True, rollback=False, raw=False, other_snapshots=False, no_send=False):
+        """sync this dataset's snapshots to target_dataset, while also thinning out old snapshots along the way."""
 
         #determine start snapshot (the first snapshot after the common snapshot)
         target_dataset.debug("Determining start snapshot")
         common_snapshot=self.find_common_snapshot(target_dataset)
+
         if not common_snapshot:
-            #start from beginning
-            start_snapshot=self.our_snapshots[0]
+            if not self.snapshots:
+                start_snapshot=None
+            else:
+                #start from beginning
+                start_snapshot=self.snapshots[0]
+
+                if not start_snapshot.is_ours() and not other_snapshots:
+                    # try to start at a snapshot thats ours
+                    start_snapshot=self.find_next_snapshot(start_snapshot, other_snapshots)
         else:
-            #roll target back to common snapshot
-            if rollback:
-                target_dataset.find_snapshot(common_snapshot).rollback()
-            start_snapshot=self.find_our_next_snapshot(common_snapshot)
+            start_snapshot=self.find_next_snapshot(common_snapshot, other_snapshots)
+
+
+        #make target snapshot list the same as source, by adding virtual non-existing ones to the list.
+        target_dataset.debug("Creating virtual target snapshots")
+        source_snapshot=start_snapshot
+        while source_snapshot:
+            #create virtual target snapshot
+            virtual_snapshot=ZfsDataset(target_dataset.zfs_node, target_dataset.filesystem_name+"@"+source_snapshot.snapshot_name,force_exists=False)
+            target_dataset.snapshots.append(virtual_snapshot)
+            source_snapshot=self.find_next_snapshot(source_snapshot, other_snapshots)
+
+
+        #now let thinner decide what we want on both sides as final state (after all transfers are done)
+        self.debug("Create thinning list")
+        if self.our_snapshots:
+            (source_keeps, source_obsoletes)=self.thin(keeps=[self.our_snapshots[-1]])
+        else:
+            source_keeps=[]
+            source_obsoletes=[]
+
+        if target_dataset.our_snapshots:
+            (target_keeps, target_obsoletes)=target_dataset.thin(keeps=[target_dataset.our_snapshots[-1]])
+        else:
+            target_keeps=[]
+            target_obsoletes=[]
+
+
+        #on source: destroy all obsoletes before common. but after common only delete snapshots that are obsolete on both sides.
+        before_common=True
+        for source_snapshot in self.snapshots:
+            if not common_snapshot or source_snapshot.snapshot_name==common_snapshot.snapshot_name:
+                before_common=False
+                #never destroy common snapshot
+            else:
+                target_snapshot=target_dataset.find_snapshot(source_snapshot)
+                if (source_snapshot in source_obsoletes) and (before_common or (target_snapshot in target_obsoletes)):
+                    source_snapshot.destroy()
+
+
+        #on target: destroy everything thats obsolete, except common_snapshot
+        for target_snapshot in target_dataset.snapshots:
+            if (not common_snapshot or target_snapshot.snapshot_name!=common_snapshot.snapshot_name) and (target_snapshot in target_obsoletes):
+                if target_snapshot.exists:
+                    target_snapshot.destroy()
+
+
+        #now actually transfer the snapshots, if we want
+        if no_send:
+            return
+
 
         #resume?
         resume_token=None
@@ -1058,46 +1154,20 @@ class ZfsDataset():
                 resume_token=None
 
 
-        #create virtual target snapshots
-        target_dataset.debug("Creating virtual target snapshots")
-        source_snapshot=start_snapshot
-        while source_snapshot:
-            #create virtual target snapshot
-            virtual_snapshot=ZfsDataset(target_dataset.zfs_node, target_dataset.filesystem_name+"@"+source_snapshot.snapshot_name,force_exists=False)
-            target_dataset.snapshots.append(virtual_snapshot)
-            source_snapshot=self.find_our_next_snapshot(source_snapshot)
+        #roll target back to common snapshot on target?
+        if common_snapshot and rollback:
+            target_dataset.find_snapshot(common_snapshot).rollback()
 
-        #now let thinner decide what we want on both sides as final state (after transfers are done)
-        self.debug("Create thinning list")
-        (source_keeps, source_obsoletes)=self.thin(keeps=[self.our_snapshots[-1]])
-        (target_keeps, target_obsoletes)=target_dataset.thin(keeps=[target_dataset.our_snapshots[-1]])
 
-        #stuff that is before common snapshot can be deleted rightaway
-        if common_snapshot:
-            for source_snapshot in self.our_snapshots:
-                if source_snapshot.snapshot_name==common_snapshot.snapshot_name:
-                    break
-
-                if source_snapshot in source_obsoletes:
-                    source_snapshot.destroy()
-
-            for target_snapshot in target_dataset.our_snapshots:
-                if target_snapshot.snapshot_name==common_snapshot.snapshot_name:
-                    break
-
-                if target_snapshot in target_obsoletes:
-                    target_snapshot.destroy()
-
-        #now send/destroy the rest off the source
+        #now actually the snapshots
         prev_source_snapshot=common_snapshot
-        prev_target_snapshot=target_dataset.find_snapshot(common_snapshot) 
         source_snapshot=start_snapshot
         while source_snapshot:
-            target_snapshot=target_dataset.find_snapshot(source_snapshot) #virtual
+            target_snapshot=target_dataset.find_snapshot(source_snapshot) #still virtual
 
             #does target actually want it?
-            if target_snapshot in target_keeps:
-                ( allowed_filter_properties, allowed_set_properties ) = self.get_allowed_properties(filter_properties, set_properties)
+            if target_snapshot not in target_obsoletes:
+                ( allowed_filter_properties, allowed_set_properties ) = self.get_allowed_properties(filter_properties, set_properties) #NOTE: should we let transfer_snapshot handle this?
                 source_snapshot.transfer_snapshot(target_snapshot, prev_snapshot=prev_source_snapshot, show_progress=show_progress, resume=resume,  filter_properties=allowed_filter_properties, set_properties=allowed_set_properties, ignore_recv_exit_code=ignore_recv_exit_code, resume_token=resume_token, raw=raw)
                 resume_token=None
 
@@ -1110,29 +1180,26 @@ class ZfsDataset():
                         prev_source_snapshot.release()
                     target_dataset.find_snapshot(prev_source_snapshot).release()
 
-                #we may destroy the previous source snapshot now, if we dont want it anymore
-                if prev_source_snapshot and (prev_source_snapshot not in source_keeps):
+                # we may now destroy the previous source snapshot if its obsolete
+                if prev_source_snapshot in source_obsoletes:
                     prev_source_snapshot.destroy()
 
-                if prev_target_snapshot and (prev_target_snapshot not in target_keeps):
+                # destroy the previous target snapshot if obsolete (usually this is only the common_snapshot, the rest was already destroyed or will not be send)
+                prev_target_snapshot=target_dataset.find_snapshot(common_snapshot)
+                if prev_target_snapshot in target_obsoletes:
                     prev_target_snapshot.destroy()
 
                 prev_source_snapshot=source_snapshot
-                prev_target_snapshot=target_snapshot
             else:
                 source_snapshot.debug("skipped (target doesnt need it)")
                 #was it actually a resume?
                 if resume_token:
                     target_dataset.debug("aborting resume, since we dont want that snapshot anymore")
                     target_dataset.abort_resume()
-                    resume_token=None   
-
-                #destroy it if we also dont want it anymore:
-                if source_snapshot not in source_keeps:
-                    source_snapshot.destroy()
+                    resume_token=None
 
 
-            source_snapshot=self.find_our_next_snapshot(source_snapshot)
+            source_snapshot=self.find_next_snapshot(source_snapshot, other_snapshots)
 
 
 
@@ -1140,7 +1207,7 @@ class ZfsDataset():
 class ZfsNode(ExecuteNode):
     """a node that contains zfs datasets. implements global (systemwide/pool wide) zfs commands"""
 
-    def __init__(self, backup_name, zfs_autobackup, ssh_to=None, readonly=False, description="", debug_output=False, thinner=Thinner()):
+    def __init__(self, backup_name, zfs_autobackup, ssh_config=None, ssh_to=None, readonly=False, description="", debug_output=False, thinner=Thinner()):
         self.backup_name=backup_name
         if not description:
             self.description=ssh_to
@@ -1149,6 +1216,9 @@ class ZfsNode(ExecuteNode):
 
         self.zfs_autobackup=zfs_autobackup #for logging
 
+        if ssh_config:
+            self.verbose("Using custom SSH config: {}".format(ssh_config))
+
         if ssh_to:
             self.verbose("Datasets on: {}".format(ssh_to))
         else:
@@ -1164,7 +1234,7 @@ class ZfsNode(ExecuteNode):
         self.thinner=thinner
 
 
-        ExecuteNode.__init__(self, ssh_to=ssh_to, readonly=readonly, debug_output=debug_output)
+        ExecuteNode.__init__(self, ssh_config=ssh_config, ssh_to=ssh_to, readonly=readonly, debug_output=debug_output)
 
 
     def reset_progress(self):
@@ -1172,9 +1242,9 @@ class ZfsNode(ExecuteNode):
         self._progress_total_bytes=0
         self._progress_start_time=time.time()
 
-    def _parse_stderr_pipe(self, line, hide_errors):
-        """try to parse progress output of a piped zfs recv -Pv """
 
+    def parse_zfs_progress(self, line, hide_errors, prefix):
+        """try to parse progress output of zfs recv -Pv, and dont show it as error to the user """
 
         #is it progress output?
         progress_fields=line.rstrip().split("\t")
@@ -1182,10 +1252,11 @@ class ZfsNode(ExecuteNode):
         if (line.find("nvlist version")==0 or
             line.find("resume token contents")==0 or
             len(progress_fields)!=1 or
-            line.find("skipping ")==0):
- 
+            line.find("skipping ")==0 or
+            re.match("send from .*estimated size is ", line)):
+
                 #always output for debugging offcourse
-                self.debug("STDERR|> "+line.rstrip())
+                self.debug(prefix+line.rstrip())
 
                 #actual usefull info
                 if len(progress_fields)>=3:
@@ -1207,15 +1278,18 @@ class ZfsNode(ExecuteNode):
 
                 return
 
-            # #is it progress output?
-            # if progress_output.find("nv")
 
-
-        #normal output without progress stuff
+        #still do the normal stderr output handling
         if hide_errors:
-            self.debug("STDERR|> "+line.rstrip())
+            self.debug(prefix+line.rstrip())
         else:
-            self.error("STDERR|> "+line.rstrip())
+            self.error(prefix+line.rstrip())
+
+    def _parse_stderr_pipe(self, line, hide_errors):
+        self.parse_zfs_progress(line, hide_errors, "STDERR|> ")
+
+    def _parse_stderr(self, line, hide_errors):
+        self.parse_zfs_progress(line, hide_errors, "STDERR > ")
 
     def verbose(self,txt):
         self.zfs_autobackup.verbose("{} {}".format(self.description, txt))
@@ -1231,23 +1305,21 @@ class ZfsNode(ExecuteNode):
         return(self.backup_name+"-"+time.strftime("%Y%m%d%H%M%S"))
 
 
-    def consistent_snapshot(self, datasets, snapshot_name, allow_empty=True):
+    def consistent_snapshot(self, datasets, snapshot_name, min_changed_bytes):
         """create a consistent (atomic) snapshot of specified datasets, per pool.
 
-        allow_empty: Allow empty snapshots. (compared to our latest snapshot)
         """
 
         pools={}
 
         #collect snapshots that we want to make, per pool
         for dataset in datasets:
-            if not allow_empty:
-                if not dataset.is_changed_ours:
-                    dataset.verbose("No changes since {}".format(dataset.our_snapshots[-1].snapshot_name))
-                    continue
+            if not dataset.is_changed_ours(min_changed_bytes):
+                dataset.verbose("No changes since {}".format(dataset.our_snapshots[-1].snapshot_name))
+                continue
 
             snapshot=ZfsDataset(dataset.zfs_node, dataset.name+"@"+snapshot_name)
-  
+
             pool=dataset.split_path()[0]
             if not pool in pools:
                 pools[pool]=[]
@@ -1255,8 +1327,7 @@ class ZfsNode(ExecuteNode):
             pools[pool].append(snapshot)
 
             #add snapshot to cache (also usefull in testmode)
-            dataset.snapshots.append(snapshot)
-
+            dataset.snapshots.append(snapshot) #NOTE: this will trigger zfs list
 
         if not pools:
             self.verbose("No changes anywhere: not creating snapshots.")
@@ -1324,8 +1395,9 @@ class ZfsAutobackup:
     def __init__(self):
 
         parser = argparse.ArgumentParser(
-            description='ZFS autobackup '+VERSION,
+            description=HEADER,
             epilog='When a filesystem fails, zfs_backup will continue and report the number of failures at that end. Also the exit code will indicate the number of failures.')
+        parser.add_argument('--ssh-config', default=None, help='Custom ssh client config')
         parser.add_argument('--ssh-source', default=None, help='Source host to get backup from. (user@hostname) Default %(default)s.')
         parser.add_argument('--ssh-target', default=None, help='Target host to push backup to. (user@hostname) Default  %(default)s.')
         parser.add_argument('--keep-source', type=str, default="10,1d1w,1w1m,1m1y", help='Thinning schedule for old source snapshots. Default: %(default)s')
@@ -1334,17 +1406,18 @@ class ZfsAutobackup:
         parser.add_argument('backup_name',    help='Name of the backup (you should set the zfs property "autobackup:backup-name" to true on filesystems you want to backup')
         parser.add_argument('target_path',    help='Target ZFS filesystem')
 
-        parser.add_argument('--no-snapshot', action='store_true', help='dont create new snapshot (usefull for finishing uncompleted backups, or cleanups)')
-        #Not appliciable anymore, version 3 alreadhy does optimal cleaning
-        # parser.add_argument('--no-send', action='store_true', help='dont send snapshots (usefull to only do a cleanup)')
-        parser.add_argument('--allow-empty', action='store_true', help='if nothing has changed, still create empty snapshots.')
+        parser.add_argument('--other-snapshots', action='store_true', help='Send over other snapshots as well, not just the ones created by this tool.')
+        parser.add_argument('--no-snapshot', action='store_true', help='Dont create new snapshots (usefull for finishing uncompleted backups, or cleanups)')
+        parser.add_argument('--no-send', action='store_true', help='Dont send snapshots (usefull for cleanups, or if you want a serperate send-cronjob)')
+        parser.add_argument('--min-change', type=int, default=200000, help='Number of bytes written after which we consider a dataset changed (default %(default)s)')
+        parser.add_argument('--allow-empty', action='store_true', help='If nothing has changed, still create empty snapshots. (same as --min-change=0)')
         parser.add_argument('--ignore-replicated', action='store_true',  help='Ignore datasets that seem to be replicated some other way. (No changes since lastest snapshot. Usefull for proxmox HA replication)')
         parser.add_argument('--no-holds', action='store_true',  help='Dont lock snapshots on the source. (Usefull to allow proxmox HA replication to switches nodes)')
         #not sure if this ever was usefull:
         # parser.add_argument('--ignore-new', action='store_true',  help='Ignore filesystem if there are already newer snapshots for it on the target (use with caution)')
 
-        parser.add_argument('--resume', action='store_true', help='support resuming of interrupted transfers by using the zfs extensible_dataset feature (both zpools should have it enabled) Disadvantage is that you need to use zfs recv -A if another snapshot is created on the target during a receive. Otherwise it will keep failing.')
-        parser.add_argument('--strip-path', default=0, type=int, help='number of directory to strip from path (use 1 when cloning zones between 2 SmartOS machines)')
+        parser.add_argument('--resume', action='store_true', help='Support resuming of interrupted transfers by using the zfs extensible_dataset feature (both zpools should have it enabled) Disadvantage is that you need to use zfs recv -A if another snapshot is created on the target during a receive. Otherwise it will keep failing.')
+        parser.add_argument('--strip-path', default=0, type=int, help='Number of directory to strip from path (use 1 when cloning zones between 2 SmartOS machines)')
         # parser.add_argument('--buffer', default="",  help='Use mbuffer with specified size to speedup zfs transfer. (e.g. --buffer 1G) Will also show nice progress output.')
 
 
@@ -1375,6 +1448,9 @@ class ZfsAutobackup:
         if self.args.test:
             self.args.verbose=True
 
+        if args.allow_empty:
+            args.min_change=0
+
         self.log=Log(show_debug=self.args.debug, show_verbose=self.args.verbose)
 
 
@@ -1392,6 +1468,9 @@ class ZfsAutobackup:
         self.log.verbose("#### "+title)
 
     def run(self):
+
+        self.verbose (HEADER)
+
         if self.args.test:
             self.verbose("TEST MODE - SIMULATING WITHOUT MAKING ANY CHANGES")
 
@@ -1399,14 +1478,14 @@ class ZfsAutobackup:
 
         description="[Source]"
         source_thinner=Thinner(self.args.keep_source)
-        source_node=ZfsNode(self.args.backup_name, self, ssh_to=self.args.ssh_source, readonly=self.args.test, debug_output=self.args.debug_output, description=description, thinner=source_thinner)
+        source_node=ZfsNode(self.args.backup_name, self, ssh_config=self.args.ssh_config, ssh_to=self.args.ssh_source, readonly=self.args.test, debug_output=self.args.debug_output, description=description, thinner=source_thinner)
         source_node.verbose("Send all datasets that have 'autobackup:{}=true' or 'autobackup:{}=child'".format(self.args.backup_name, self.args.backup_name))
 
         self.verbose("")
 
         description="[Target]"
         target_thinner=Thinner(self.args.keep_target)
-        target_node=ZfsNode(self.args.backup_name, self, ssh_to=self.args.ssh_target, readonly=self.args.test, debug_output=self.args.debug_output, description=description, thinner=target_thinner)
+        target_node=ZfsNode(self.args.backup_name, self, ssh_config=self.args.ssh_config, ssh_to=self.args.ssh_target, readonly=self.args.test, debug_output=self.args.debug_output, description=description, thinner=target_thinner)
         target_node.verbose("Receive datasets under: {}".format(self.args.target_path))
 
         self.set_title("Selecting")
@@ -1423,7 +1502,7 @@ class ZfsAutobackup:
         else:
             self.set_title("Filtering already replicated filesystems")
             for selected_source_dataset in selected_source_datasets:
-                if selected_source_dataset.is_changed():
+                if selected_source_dataset.is_changed(self.args.min_change):
                     source_datasets.append(selected_source_dataset)
                 else:
                     selected_source_dataset.verbose("Ignoring, already replicated")
@@ -1431,10 +1510,14 @@ class ZfsAutobackup:
 
         if not self.args.no_snapshot:
             self.set_title("Snapshotting")
-            source_node.consistent_snapshot(source_datasets, source_node.new_snapshotname(), allow_empty=self.args.allow_empty)
+            source_node.consistent_snapshot(source_datasets, source_node.new_snapshotname(), min_changed_bytes=self.args.min_change)
 
 
-        self.set_title("Transferring")
+
+        if self.args.no_send:        
+            self.set_title("Thinning")
+        else:
+            self.set_title("Sending and thinning")
 
         if self.args.filter_properties:
             filter_properties=self.args.filter_properties.split(",")
@@ -1450,7 +1533,7 @@ class ZfsAutobackup:
             filter_properties.append("refreservation")
 
         if self.args.clear_mountpoint:
-            set_properties.append( "canmount=noauto"  )
+            set_properties.append("canmount=noauto")
 
         fail_count=0
         for source_dataset in source_datasets:
@@ -1461,10 +1544,10 @@ class ZfsAutobackup:
                 target_dataset=ZfsDataset(target_node, target_name)
 
                 #ensure parents exists
-                if not target_dataset.parent.exists:
+                if not self.args.no_send and not target_dataset.parent.exists:
                     target_dataset.parent.create_filesystem(parents=True)
 
-                source_dataset.sync_snapshots(target_dataset, show_progress=self.args.progress, resume=self.args.resume, filter_properties=filter_properties, set_properties=set_properties, ignore_recv_exit_code=self.args.ignore_transfer_errors, source_holds= not self.args.no_holds, rollback=self.args.rollback, raw=self.args.raw)
+                source_dataset.sync_snapshots(target_dataset, show_progress=self.args.progress, resume=self.args.resume, filter_properties=filter_properties, set_properties=set_properties, ignore_recv_exit_code=self.args.ignore_transfer_errors, source_holds= not self.args.no_holds, rollback=self.args.rollback, raw=self.args.raw, other_snapshots=self.args.other_snapshots, no_send=self.args.no_send)
             except Exception as e:
                 fail_count=fail_count+1
                 source_dataset.error("DATASET FAILED: "+str(e))
@@ -1489,5 +1572,3 @@ class ZfsAutobackup:
 if __name__ == "__main__":
     zfs_autobackup=ZfsAutobackup()
     sys.exit(zfs_autobackup.run())
-
-

release

@@ -15,3 +15,4 @@ source token
 python3 -m twine check dist/*
 python3 -m twine upload dist/*
 
+git push --tags

setup.py

@@ -2,7 +2,7 @@ import setuptools
 import bin.zfs_autobackup
 import os
 
-os.system("git tag -m'' -a v{}".format(bin.zfs_autobackup.VERSION))
+os.system("git tag -m ' ' -a v{}".format(bin.zfs_autobackup.VERSION))
 
 with open("README.md", "r") as fh:
     long_description = fh.read()