From e264ddba913764298fc9ae1c7d03349a4ad5042c Mon Sep 17 00:00:00 2001 From: Edwin Eefting Date: Fri, 7 Jan 2022 10:52:39 +0100 Subject: [PATCH] update --- Piping.md | 56 +++++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 48 insertions(+), 8 deletions(-) diff --git a/Piping.md b/Piping.md index 6b09c5e..98b020b 100644 --- a/Piping.md +++ b/Piping.md @@ -1,22 +1,62 @@ -## Transfer buffering, compression and rate limiting. +During transfer the data will be piped from the source to the target. -If you're transferring over a slow link it might be useful to use `--compress=zstd-fast`. This will compress the data before sending, so it uses less bandwidth. An alternative to this is to use --zfs-compressed: This will transfer blocks that already have compression intact. (--compress will usually compress much better but uses much more resources. --zfs-compressed uses the least resources, but can be a disadvantage if you want to use a different compression method on the target.) +Its possible to add certain operations to this pipe. -You can also limit the datarate by using the `--rate` option. +## Transfer buffering -The `--buffer` option might also help since it acts as an IO buffer: zfs send can vary wildly between completely idle and huge bursts of data. When zfs send is idle, the buffer will continue transferring data over the slow link. + +The `--buffer` option might help since it acts as an IO buffer: zfs send can vary wildly between completely idle and huge bursts of data. When zfs send is idle, the buffer will continue transferring data to the other side. + +This needs the mbuffer command on both sides. +## Compression + +If you're transferring over a slow link it might be useful to use `--compress`. This will compress the data before sending, so it uses less bandwidth. + +An alternative to this is to use `--zfs-compressed`: This will transfer blocks that already have compression intact. Also it + +* `--compress` will usually compress much better but uses much more resources. +* ` --zfs-compressed` uses the least resources, but can be a disadvantage if you want to use a different compression method on the target. + +Dont use both options at the same time, since its probably wont help. + +By default `--compress` uses pigz-fast. Use `--compress=...` to select a specific compressor: + +* `pigz-fast`: Uses pigz -3 +* `pigz-slow`: Uses pigz -9 +* `gzip`: Uses gzip -3 and zcat. +* `zstd-fast`: Uses zstdmt -3 +* `zstd-slow`: Uses zstdmt -19 +* `zstd-adapt`: Uses zstdmt --adapt +* `xz`: Uses xz +* `lzo`: Uses lzop +* `lz4`: Uses lz4 + +Offcourse the specific compressors need to be installed on both sides. +## Rate limiting + +If you want to limit the datarate, try using the `--rate` option. This is usefull to not saturate a slow uplink or do reduce IO load. + +## Custom pipes It's also possible to add custom send or receive pipes with `--send-pipe` and `--recv-pipe`. -These options all work together and the buffer on the receiving side is only added if appropriate. When all options are active: +This way you can pipe the data through and custom compressor or command you like. -On the sending side: +## Putting it all together + +These options all work together, when all options are active: + +Pipe on the the sending side: ``` -zfs send -> send buffer -> custom send pipes -> compression -> transfer rate limiter +zfs send | buffer | custom send pipes | compression | transfer rate limiter | ssh ``` On the receiving side: ``` -decompression -> custom recv pipes -> buffer -> zfs recv +decompression | custom recv pipes | buffer | zfs recv ``` + +The buffer on the receiving side is only added if its on a different host. + +Also zfs-autobackup will warn you if you do something useless, like using --compress without ssh.