rc1

.github/workflows/regression.yml

@@ -17,7 +17,7 @@ jobs:
 
 
       - name: Prepare
-        run: sudo apt update && sudo apt install zfsutils-linux lzop pigz zstd gzip xz-utils lz4 && sudo -H pip3 install coverage unittest2 mock==3.0.5 coveralls
+        run: sudo apt update && sudo apt install zfsutils-linux lzop pigz zstd gzip xz-utils lz4 mbuffer && sudo -H pip3 install coverage unittest2 mock==3.0.5 coveralls
 
 
       - name: Regression test
@@ -39,7 +39,7 @@ jobs:
 
 
       - name: Prepare
-        run: sudo apt update && sudo apt install zfsutils-linux python3-setuptools lzop pigz zstd gzip xz-utils liblz4-tool && sudo -H pip3 install coverage unittest2 mock==3.0.5 coveralls
+        run: sudo apt update && sudo apt install zfsutils-linux python3-setuptools lzop pigz zstd gzip xz-utils liblz4-tool mbuffer && sudo -H pip3 install coverage unittest2 mock==3.0.5 coveralls
 
 
       - name: Regression test
@@ -64,7 +64,7 @@ jobs:
           python-version: '2.x'
 
       - name: Prepare
-        run: sudo apt update && sudo apt install zfsutils-linux python-setuptools lzop pigz zstd gzip xz-utils liblz4-tool && sudo -H pip install coverage unittest2 mock==3.0.5 coveralls colorama
+        run: sudo apt update && sudo apt install zfsutils-linux python-setuptools lzop pigz zstd gzip xz-utils liblz4-tool mbuffer && sudo -H pip install coverage unittest2 mock==3.0.5 coveralls colorama
 
       - name: Regression test
         run: sudo -E ./tests/run_tests

README.md

@@ -5,7 +5,7 @@
 
 ## Introduction
 
-This is a tool I wrote to make replicating ZFS datasets easy and reliable.
+ZFS-autobackup tries to be the most reliable and easiest to use tool, while having all the features.
 
 You can either use it as a **backup** tool, **replication** tool or **snapshot** tool.
 
@@ -13,12 +13,10 @@ You can select what to backup by setting a custom `ZFS property`. This makes it
 
 Other settings are just specified on the commandline: Simply setup and test your zfs-autobackup command and  fix all the issues you might encounter. When you're done you can just copy/paste your command to a cron or script.
 
-Since its using ZFS commands, you can see what its actually doing by specifying `--debug`. This also helps a lot if you run into some strange problem or error. You can just copy-paste the command that fails and play around with it on the commandline. (something I missed in other tools)
+Since its using ZFS commands, you can see what it's actually doing by specifying `--debug`. This also helps a lot if you run into some strange problem or error. You can just copy-paste the command that fails and play around with it on the commandline. (something I missed in other tools)
 
 An important feature thats missing from other tools is a reliable `--test` option: This allows you to see what zfs-autobackup will do and tune your parameters. It will do everything, except make changes to your system.
 
-zfs-autobackup tries to be the easiest to use backup tool for zfs, with the most features.
-
 ## Features
 
 * Works across operating systems: Tested with **Linux**, **FreeBSD/FreeNAS** and **SmartOS**.
@@ -34,6 +32,9 @@ zfs-autobackup tries to be the easiest to use backup tool for zfs, with the most
 * Can be scheduled via a simple cronjob or run directly from commandline.
 * Supports resuming of interrupted transfers.
 * ZFS encryption support: Can decrypt / encrypt or even re-encrypt datasets during transfer.
+* Supports sending with compression. (Using pigz, zstd etc)
+* IO buffering to speed up transfer.
+* Bandwidth rate limiting.
 * Multiple backups from and to the same datasets are no problem.
 * Creates the snapshot before doing anything else. (assuring you at least have a snapshot if all else fails)
 * Checks everything but tries continue on non-fatal errors when possible. (Reports error-count when done)
@@ -44,10 +45,11 @@ zfs-autobackup tries to be the easiest to use backup tool for zfs, with the most
 * Automatic resuming of failed transfers.
 * Can continue from existing common snapshots. (e.g. easy migration)
 * Gracefully handles datasets that no longer exist on source.
+* Support for ZFS sending/receiving through custom pipes.
 * Easy installation:
-  * Just install zfs-autobackup via pip, or download it manually.
+  * Just install zfs-autobackup via pip.
   * Only needs to be installed on one side.
-  * Written in python and uses zfs-commands, no 3rd party dependency's or libraries needed.
+  * Written in python and uses zfs-commands, no special 3rd party dependency's or compiled libraries needed.
   * No separate config files or properties. Just one zfs-autobackup command you can copy/paste in your backup script.
 
 ## Installation
@@ -396,6 +398,25 @@ Note 2: Decide what you want at an early stage: If you change the --encrypt or -
 
 I'll add some tips when the issues start to get in on github. :)
 
+## Transfer buffering, compression and rate limiting.
+
+If you're transferring over a slow link it might be useful to use `--compress=zstd-fast`. This will compres the data before sending, so it uses less bandwidth.
+
+You can also limit the datarate by using the `--rate` option.
+
+The `--buffer` option might also help since it acts as an IO buffer: zfs send can vary wildly between completely idle and huge bursts of data. When zfs send is idle, the buffer will continue transferring data over the slow link.
+
+It's also possible to add custom send or receive pipes with `--send-pipe` and `--recv-pipe`.
+
+These options all work together and the buffer on the receiving side is only added if appropriate. When all options are active:
+
+#### On the sending side:
+
+zfs send -> send buffer -> custom send pipes -> compression -> transfer rate limiter
+
+#### On the receiving side:
+decompression -> custom recv pipes -> buffer -> zfs recv
+
 ## Tips
 
 * Use ```--debug``` if something goes wrong and you want to see the commands that are executed. This will also stop at the first error.
@@ -408,6 +429,8 @@ I'll add some tips when the issues start to get in on github. :)
 
 If you have a large number of datasets its important to keep the following tips in mind.
 
+Also it might help to use the --buffer option to add IO buffering during the data transfer. This might speed up things since it smooths out sudden IO bursts that are frequent during a zfs send or recv.
+
 #### Some statistics
 
 To get some idea of how fast zfs-autobackup is, I did some test on my laptop, with a SKHynix_HFS512GD9TNI-L2B0B disk. I'm using zfs 2.0.2.  
@@ -468,17 +491,31 @@ Look in man ssh_config for many more options.
 (NOTE: Quite a lot has changed since the current stable version 3.0. The page your are viewing is for upcoming version 3.1 which is still in beta.)
 
 ```console
-usage: zfs-autobackup [-h] [--ssh-config CONFIG-FILE] [--ssh-source USER@HOST] [--ssh-target USER@HOST] [--keep-source SCHEDULE] [--keep-target SCHEDULE] [--other-snapshots] [--no-snapshot] [--no-send]
-                   [--no-thinning] [--no-holds] [--min-change BYTES] [--allow-empty] [--ignore-replicated] [--strip-path N] [--clear-refreservation] [--clear-mountpoint] [--filter-properties PROPERY,...]
-                   [--set-properties PROPERTY=VALUE,...] [--rollback] [--destroy-incompatible] [--destroy-missing SCHEDULE] [--ignore-transfer-errors] [--decrypt] [--encrypt] [--test] [--verbose] [--debug]
-                   [--debug-output] [--progress] [--send-pipe COMMAND] [--recv-pipe COMMAND]
+usage: zfs-autobackup [-h] [--ssh-config CONFIG-FILE] [--ssh-source USER@HOST]
+                      [--ssh-target USER@HOST] [--keep-source SCHEDULE]
+                      [--keep-target SCHEDULE] [--other-snapshots]
+                      [--no-snapshot] [--no-send] [--no-thinning] [--no-holds]
+                      [--min-change BYTES] [--allow-empty]
+                      [--ignore-replicated] [--strip-path N]
+                      [--clear-refreservation] [--clear-mountpoint]
+                      [--filter-properties PROPERY,...]
+                      [--set-properties PROPERTY=VALUE,...] [--rollback]
+                      [--destroy-incompatible] [--destroy-missing SCHEDULE]
+                      [--ignore-transfer-errors] [--decrypt] [--encrypt]
+                      [--test] [--verbose] [--debug] [--debug-output]
+                      [--progress] [--send-pipe COMMAND] [--recv-pipe COMMAND]
+                      [--compress TYPE] [--rate DATARATE] [--buffer SIZE]
                       backup-name [target-path]
 
-zfs-autobackup v3.1-beta3 - Copyright 2020 E.H.Eefting (edwin@datux.nl)
+zfs-autobackup v3.1-beta6 - (c)2021 E.H.Eefting (edwin@datux.nl)
 
 positional arguments:
-  backup-name           Name of the backup (you should set the zfs property "autobackup:backup-name" to true on filesystems you want to backup
-  target-path           Target ZFS filesystem (optional: if not specified, zfs-autobackup will only operate as snapshot-tool on source)
+  backup-name           Name of the backup (you should set the zfs property
+                        "autobackup:backup-name" to true on filesystems you
+                        want to backup
+  target-path           Target ZFS filesystem (optional: if not specified,
+                        zfs-autobackup will only operate as snapshot-tool on
+                        source)
 
 optional arguments:
   -h, --help            show this help message and exit
@@ -489,41 +526,78 @@ optional arguments:
   --ssh-target USER@HOST
                         Target host to push backup to.
   --keep-source SCHEDULE
-                        Thinning schedule for old source snapshots. Default: 10,1d1w,1w1m,1m1y
+                        Thinning schedule for old source snapshots. Default:
+                        10,1d1w,1w1m,1m1y
   --keep-target SCHEDULE
-                        Thinning schedule for old target snapshots. Default: 10,1d1w,1w1m,1m1y
-  --other-snapshots     Send over other snapshots as well, not just the ones created by this tool.
-  --no-snapshot         Don't create new snapshots (useful for finishing uncompleted backups, or cleanups)
-  --no-send             Don't send snapshots (useful for cleanups, or if you want a serperate send-cronjob)
+                        Thinning schedule for old target snapshots. Default:
+                        10,1d1w,1w1m,1m1y
+  --other-snapshots     Send over other snapshots as well, not just the ones
+                        created by this tool.
+  --no-snapshot         Don't create new snapshots (useful for finishing
+                        uncompleted backups, or cleanups)
+  --no-send             Don't send snapshots (useful for cleanups, or if you
+                        want a serperate send-cronjob)
   --no-thinning         Do not destroy any snapshots.
-  --no-holds            Don't hold snapshots. (Faster. Allows you to destroy common snapshot.)
-  --min-change BYTES    Number of bytes written after which we consider a dataset changed (default 1)
-  --allow-empty         If nothing has changed, still create empty snapshots. (same as --min-change=0)
-  --ignore-replicated   Ignore datasets that seem to be replicated some other way. (No changes since lastest snapshot. Useful for proxmox HA replication)
-  --strip-path N        Number of directories to strip from target path (use 1 when cloning zones between 2 SmartOS machines)
+  --no-holds            Don't hold snapshots. (Faster. Allows you to destroy
+                        common snapshot.)
+  --min-change BYTES    Number of bytes written after which we consider a
+                        dataset changed (default 1)
+  --allow-empty         If nothing has changed, still create empty snapshots.
+                        (same as --min-change=0)
+  --ignore-replicated   Ignore datasets that seem to be replicated some other
+                        way. (No changes since lastest snapshot. Useful for
+                        proxmox HA replication)
+
+  --strip-path N        Number of directories to strip from target path (use 1
+                        when cloning zones between 2 SmartOS machines)
   --clear-refreservation
-                        Filter "refreservation" property. (recommended, safes space. same as --filter-properties refreservation)
-  --clear-mountpoint    Set property canmount=noauto for new datasets. (recommended, prevents mount conflicts. same as --set-properties canmount=noauto)
+                        Filter "refreservation" property. (recommended, safes
+                        space. same as --filter-properties refreservation)
+  --clear-mountpoint    Set property canmount=noauto for new datasets.
+                        (recommended, prevents mount conflicts. same as --set-
+                        properties canmount=noauto)
   --filter-properties PROPERY,...
-                        List of properties to "filter" when receiving filesystems. (you can still restore them with zfs inherit -S)
+                        List of properties to "filter" when receiving
+                        filesystems. (you can still restore them with zfs
+                        inherit -S)
   --set-properties PROPERTY=VALUE,...
-                        List of propererties to override when receiving filesystems. (you can still restore them with zfs inherit -S)
-  --rollback            Rollback changes to the latest target snapshot before starting. (normally you can prevent changes by setting the readonly property on the target_path to on)
+                        List of propererties to override when receiving
+                        filesystems. (you can still restore them with zfs
+                        inherit -S)
+  --rollback            Rollback changes to the latest target snapshot before
+                        starting. (normally you can prevent changes by setting
+                        the readonly property on the target_path to on)
   --destroy-incompatible
-                        Destroy incompatible snapshots on target. Use with care! (implies --rollback)
+                        Destroy incompatible snapshots on target. Use with
+                        care! (implies --rollback)
   --destroy-missing SCHEDULE
-                        Destroy datasets on target that are missing on the source. Specify the time since the last snapshot, e.g: --destroy-missing 30d
+                        Destroy datasets on target that are missing on the
+                        source. Specify the time since the last snapshot, e.g:
+                        --destroy-missing 30d
   --ignore-transfer-errors
-                        Ignore transfer errors (still checks if received filesystem exists. useful for acltype errors)
+                        Ignore transfer errors (still checks if received
+                        filesystem exists. useful for acltype errors)
   --decrypt             Decrypt data before sending it over.
   --encrypt             Encrypt data after receiving it.
-  --test                dont change anything, just show what would be done (still does all read-only operations)
+  --test                dont change anything, just show what would be done
+                        (still does all read-only operations)
   --verbose             verbose output
-  --debug               Show zfs commands that are executed, stops after an exception.
+  --debug               Show zfs commands that are executed, stops after an
+                        exception.
   --debug-output        Show zfs commands and their output/exit codes. (noisy)
-  --progress            show zfs progress output. Enabled automaticly on ttys. (use --no-progress to disable)
-  --send-pipe COMMAND   pipe zfs send output through COMMAND
-  --recv-pipe COMMAND   pipe zfs recv input through COMMAND
+  --progress            show zfs progress output. Enabled automaticly on ttys.
+                        (use --no-progress to disable)
+  --send-pipe COMMAND   pipe zfs send output through COMMAND (can be used
+                        multiple times)
+  --recv-pipe COMMAND   pipe zfs recv input through COMMAND (can be used
+                        multiple times)
+  --compress TYPE       Use compression during transfer, zstd-fast
+                        recommended. (zstd-slow, xz, pigz-fast, lz4, pigz-
+                        slow, zstd-fast, gzip, lzo)
+  --rate DATARATE       Limit data transfer rate (e.g. 128K. requires
+                        mbuffer.)
+  --buffer SIZE         Add zfs send and recv buffers to smooth out IO bursts.
+                        (e.g. 128M. requires mbuffer)
 
 Full manual at: https://github.com/psy0rz/zfs_autobackup
 ```

tests/basetest.py

@@ -1,4 +1,6 @@
 
+# To run tests as non-root, use this hack:
+# chmod 4755 /usr/sbin/zpool /usr/sbin/zfs
 
 import subprocess
 import random

tests/test_cmdpipe.py

@@ -21,7 +21,7 @@ class TestCmdPipe(unittest2.TestCase):
         p=CmdPipe(readonly=False, inp="test")
         err=[]
         out=[]
-        p.add(CmdItem(["echo", "test"], stderr_handler=lambda line: err.append(line), exit_handler=lambda exit_code: self.assertEqual(exit_code,0)))
+        p.add(CmdItem(["cat"], stderr_handler=lambda line: err.append(line), exit_handler=lambda exit_code: self.assertEqual(exit_code,0)))
         executed=p.execute(stdout_handler=lambda line: out.append(line))
 
         self.assertEqual(err, [])

tests/test_sendrecvpipes.py

@@ -47,3 +47,42 @@ class TestSendRecvPipes(unittest2.TestCase):
                     self.assertFalse(ZfsAutobackup(["test", "test_target1", "--exclude-received", "--no-holds", "--no-progress", "--compress="+compress]).run())
 
                 shelltest("zfs destroy -r test_target1/test_source1/fs1/sub")
+
+    def test_buffer(self):
+        """test different buffer configurations"""
+
+
+        with self.subTest("local local pipe"):
+            with patch('time.strftime', return_value="20101111000000"):
+                self.assertFalse(ZfsAutobackup(["test", "test_target1", "--exclude-received", "--no-holds", "--no-progress", "--buffer=1M" ]).run())
+
+            shelltest("zfs destroy -r test_target1/test_source1/fs1/sub")
+
+        with self.subTest("remote local pipe"):
+            with patch('time.strftime', return_value="20101111000000"):
+                self.assertFalse(ZfsAutobackup(["test", "test_target1", "--exclude-received", "--no-holds", "--no-progress", "--ssh-source=localhost", "--buffer=1M"]).run())
+
+            shelltest("zfs destroy -r test_target1/test_source1/fs1/sub")
+
+        with self.subTest("local remote pipe"):
+            with patch('time.strftime', return_value="20101111000000"):
+                self.assertFalse(ZfsAutobackup(["test", "test_target1",  "--exclude-received", "--no-holds", "--no-progress", "--ssh-target=localhost", "--buffer=1M"]).run())
+
+            shelltest("zfs destroy -r test_target1/test_source1/fs1/sub")
+
+        with self.subTest("remote remote pipe"):
+            with patch('time.strftime', return_value="20101111000000"):
+                self.assertFalse(ZfsAutobackup(["test", "test_target1",  "--exclude-received", "--no-holds", "--no-progress", "--ssh-source=localhost", "--ssh-target=localhost", "--buffer=1M"]).run())
+
+    def test_rate(self):
+        """test rate limit"""
+
+
+        start=time.time()
+        with patch('time.strftime', return_value="20101111000000"):
+            self.assertFalse(ZfsAutobackup(["test", "test_target1", "--exclude-received", "--no-holds", "--no-progress", "--rate=50k" ]).run())
+
+        #not a great way of verifying but it works.
+        self.assertGreater(time.time()-start, 5)
+
+

zfs_autobackup/ZfsAutobackup.py

@@ -16,7 +16,7 @@ from zfs_autobackup.ThinnerRule import ThinnerRule
 class ZfsAutobackup:
     """main class"""
 
-    VERSION = "3.1-beta6"
+    VERSION = "3.1-rc1"
     HEADER = "zfs-autobackup v{} - (c)2021 E.H.Eefting (edwin@datux.nl)".format(VERSION)
 
     def __init__(self, argv, print_arguments=True):
@@ -163,7 +163,7 @@ class ZfsAutobackup:
             self.log.error("Target should not start with a /")
             sys.exit(255)
 
-        if args.compress and not args.ssh_source and not args.ssh_target:
+        if args.compress and args.ssh_source is None and args.ssh_target is None:
             self.warning("Using compression, but transfer is local.")
 
     def verbose(self, txt):
@@ -323,6 +323,13 @@ class ZfsAutobackup:
             ret.append(ExecuteNode.PIPE)
             logger("zfs recv custom pipe   : {}".format(recv_pipe))
 
+        # IO buffer
+        if self.args.buffer:
+            #only add second buffer if its usefull. (e.g. non local transfer or other pipes active)
+            if self.args.ssh_source!=None or self.args.ssh_target!=None or self.args.recv_pipe or self.args.send_pipe or self.args.compress!=None:
+                logger("zfs recv buffer        : {}".format(self.args.buffer))
+                ret.extend(["mbuffer", "-q", "-s128k", "-m"+self.args.buffer,  ExecuteNode.PIPE ])
+
         return ret
 
     # NOTE: this method also uses self.args. args that need extra processing are passed as function parameters: