Hi all, on my laptop I’ve two nvme, one of 2Tb with linux-zfs, and one of 1Tb with freebsd. Also I’ve one external nvme on a case that I’ve formatted on zfs for using how backup.
This in my situationmarco@gentsar ~ $ zpool list -v NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT backup 928G 302G 626G - - 0% 32% 1.00x ONLINE - sda2 931G 302G 626G - - 0% 32.5% - ONLINE lpool 1.80T 292G 1.51T - - 0% 15% 1.00x ONLINE - nvme-MSI_M480_PRO_2TB_511240829130000340-part3 1.80T 292G 1.51T - - 0% 15.9% - ONLINE marco@gentsar ~ $
marco@gentsar ~ $ zfs list
NAME USED AVAIL REFER MOUNTPOINT
backup 302G 597G 384K /backup
backup/bsd 9.62G 597G 384K none
backup/bsd/freebsd-home 384K 597G 384K none
backup/bsd/freebsd-home-marco 503M 597G 503M none
backup/bsd/freebsd-root 9.13G 597G 9.13G none
backup/condivise 180G 597G 180G /backup/condivise
backup/linux 113G 597G 384K none
backup/linux/arch-home 23.9G 597G 23.9G none
backup/linux/arch-root 21.0G 597G 21.0G none
backup/linux/chimera-home 1.67G 597G 1.67G none
backup/linux/chimera-root 13.7G 597G 13.7G none
backup/linux/gentoo-root 31.6G 597G 31.6G none
backup/linux/void-home 9.43G 597G 9.43G none
backup/linux/void-root 11.4G 597G 11.4G none
lpool 293G 1.46T 96K none
lpool/condivise 178G 1.46T 178G legacy
lpool/home 46.2G 1.46T 96K none
lpool/home/arch 22.7G 1.46T 19.9G legacy
lpool/home/chimera 2.23G 1.46T 1.23G legacy
lpool/home/fedora 96K 1.46T 96K none
lpool/home/gentoo 11.9G 1.46T 10.3G legacy
lpool/home/void 9.39G 1.46T 5.37G legacy
lpool/root 68.7G 1.46T 96K none
lpool/root/arch 20.4G 1.46T 16.6G /
lpool/root/chimera 15.2G 1.46T 9.61G /
lpool/root/gentoo 24.5G 1.46T 20.6G /
lpool/root/void 8.54G 1.46T 7.98G /
for snapshot I’m using zrepl and follow its site I created this config:
marco@gentsar ~ $ cat /etc/zrepl/zrepl.yml
# This config serves as an example for a local zrepl installation that
# backups the entire zpool `system` to `backuppool/zrepl/sink`
#
# The requirements covered by this setup are described in the zrepl documentation's
# quick start section which inlines this example.
#
# CUSTOMIZATIONS YOU WILL LIKELY WANT TO APPLY:
# - adjust the name of the production pool `system` in the `filesystems` filter of jobs `snapjob` and `push_to_drive`
# - adjust the name of the backup pool `backuppool` in the `backuppool_sink` job
# - adjust the occurences of `myhostname` to the name of the system you are backing up (cannot be easily changed once you start replicating)
# - make sure the `zrepl_` prefix is not being used by any other zfs tools you might have installed (it likely isn't)
jobs:
# this job takes care of snapshot creation + pruning
- name: snapjob
type: snap
filesystems: {
"lpool/root/gentoo": true,
"lpool/home/gentoo": true,
}
# create snapshots with prefix `zrepl_` every 15 minutes
snapshotting:
type: periodic
interval: 15m
prefix: zrepl_
pruning:
keep:
# fade-out scheme for snapshots starting with `zrepl_`
# - keep all created in the last hour
# - then destroy snapshots such that we keep 24 each 1 hour apart
# - then destroy snapshots such that we keep 14 each 1 day apart
# - then destroy all older snapshots
- type: grid
grid: 1x1h(keep=all) | 24x1h | 14x1d
regex: "^zrepl_.*"
# keep all snapshots that don't have the `zrepl_` prefix
- type: regex
negate: true
regex: "^zrepl_.*"
# This job pushes to the local sink defined in job `backuppool_sink`.
# We trigger replication manually from the command line / udev rules using
# `zrepl signal wakeup push_to_drive`
- type: push
name: push_to_drive
connect:
type: local
listener_name: backup
client_identity: gentsar
filesystems: {
"lpool/root/gentoo": true,
"lpool/home/gentoo": true,
}
send:
encrypted: false
replication:
protection:
initial: guarantee_resumability
# Downgrade protection to guarantee_incremental which uses zfs bookmarks instead of zfs holds.
# Thus, when we yank out the backup drive during replication
# - we might not be able to resume the interrupted replication step because the partially received `to` snapshot of a `from`->`to` step may be pruned any time
# - but in exchange we get back the disk space allocated by `to` when we prune it
# - and because we still have the bookmarks created by `guarantee_incremental`, we can still do incremental replication of `from`->`to2` in the future
incremental: guarantee_incremental
snapshotting:
type: manual
pruning:
# no-op prune rule on sender (keep all snapshots), job `snapshot` takes care of this
keep_sender:
- type: regex
regex: ".*"
# retain
keep_receiver:
# longer retention on the backup drive, we have more space there
- type: grid
grid: 1x1h(keep=all) | 24x1h | 360x1d
regex: "^zrepl_.*"
# retain all non-zrepl snapshots on the backup drive
- type: regex
negate: true
regex: "^zrepl_.*"
# This job receives from job `push_to_drive` into `backuppool/zrepl/sink/myhostname`
- type: sink
name: backup
root_fs: "backup/linux"
serve:
type: local
listener_name: backup
This config create snapshots correctly, but I don’t understand when snapshts are moved on backup pool.
Someone use it?
I use zrepl for backups, including to a removable drive. Snapshots are never ‘moved’, they are copied.
The configuration you have posted requires a manual trigger to cause the push_to_drive job to run. To do that you’d run zrepl status, scroll down to the push_to_drive job, and press S to trigger the job.
Thanks so much, tomorrow I’ll try. But if I want to automatized? Can I do it?
Certainly, you can use any tool you like (cron, anacron, systemd timers, etc.) to run zrepl signal to trigger the job. I do this on laptops, triggering the job after the laptop comes out of suspend mode.
I’m very stupid. I’ve launch zrepl status and trigger push_to_drive, but I’ve this error:
═╗┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
║jobs ║│Job: push_to_drive │
║├──backup ║│Type: push │
║├──push_to_drive ║│ │
║└──snapjob ║│Replication: │
║ ║│ Attempt #1 │
║ ║│ Status: filesystem-error │
║ ║│ Last Run: 2025-01-25 10:40:24 +0100 CET (lasted 0s) │
║ ║│ Problem: one or more of the filesystems encountered errors │
║ ║│ Progress: [>--------------------------------------------------] 9.5 MiB / 46.3 GiB @ 0 B/s │
║ ║│ lpool/home/gentoo STEP-ERROR (step 1/1, 5.5 MiB/13.6 GiB) server error: cannot create placeholder filesystem backup/linux-root/gentsar/lpool: placeholder filesystem encryption handling │
║ ║│ lpool/root/gentoo STEP-ERROR (step 1/1, 4.0 MiB/32.6 GiB) server error: cannot create placeholder filesystem backup/linux-root/gentsar/lpool: placeholder filesystem encryption handling │
║ ║│ │
║ ║│Pruning Sender: │
║ ║│ Status: ExecErr │
║ ║│ Progress: [>] 0/0 snapshots │
║ ║│ lpool/home/gentoo ERROR: replication cursor bookmark does not exist (one successful replication is required before pruning works) │
║ ║│ │
║ ║│ lpool/root/gentoo ERROR: replication cursor bookmark does not exist (one successful replication is required before pruning works) │
║ ║│ │
║ ║│ │
║ ║│Pruning Receiver: │
║ ║│ Status: Done │
║ ║│ nothing to do │
║ ║│ │
║ ║│Snapshotting: │
║ ║│ Type: manual │
║ ║│ <no details available> │
║ ║│
on /var/log/zrepl.log I read this:
2025-01-25T10:51:55+01:00 [ERRO][push_to_drive][pruning][jHmm$EuX7$EuX7.LHnX.wIhD.ABEk]: : plan error, skipping filesystem prune_side="sender" err="replication cursor bookmark does not exist (one successful replication is required before pruning works)" fs="lpool/root/gentoo" orig_err_type="*errors.fundamental"
maybe problems are that my datasets aren’t encrypted. But I don’t understand how using send without encryption
It would probably be best to ask these questions in the zrepl repository or its Matrix channel.
this is my zrepl config that pushes encrypted backups to an external usb drive:
[root@endeavour ~]# cat /etc/zrepl/zrepl.yml
# This config serves as an example for a local zrepl installation that
# backups the entire zpool `system` to `backuppool/zrepl/sink`
#
# The requirements covered by this setup are described in the zrepl documentation's
# quick start section which inlines this example.
#
# CUSTOMIZATIONS YOU WILL LIKELY WANT TO APPLY:
# - adjust the name of the production pool `system` in the `filesystems` filter of jobs `snapjob` and `push_to_derive`
# - adjust the name of the backup pool `backuppool` in the `backuppool_sink` job
# - adjust the occurences of `myhostname` to the name of the system you are backing up (cannot be easily changed once you start replicating)
# - make sure the `zrepl_` prefix is not being used by any other zfs tools you might have installed (it likely isn't)
global:
logging:
- type: "stdout"
level: "warn"
format: "human"
- type: "syslog"
level: "warn"
format: "human"
jobs:
# this job takes care of snapshot creation + pruning
- name: snapjob
type: snap
filesystems: {
"rust1<": true,
}
# create snapshots with prefix `zrepl_` every 15 minutes
snapshotting:
type: periodic
interval: 15m
prefix: zrepl_
pruning:
keep:
# fade-out scheme for snapshots starting with `zrepl_`
# - keep all created in the last hour
# - then destroy snapshots such that we keep 24 each 1 hour apart
# - then destroy snapshots such that we keep 14 each 1 day apart
# - then destroy all older snapshots
- type: grid
grid: 1x1h(keep=all) | 24x1h | 14x1d
regex: "^zrepl_.*"
# keep all snapshots that don't have the `zrepl_` prefix
- type: regex
negate: true
regex: "^zrepl_.*"
# This job pushes to the local sink defined in job `backuppool_sink`.
# We trigger replication manually from the command line / udev rules using
# `zrepl signal wakeup push_to_drive`
- type: push
name: push_to_drive
connect:
type: local
listener_name: backuppool_sink
client_identity: endeavour
filesystems: {
"rust1/crypt<": true
}
send:
encrypted: true
replication:
protection:
initial: guarantee_resumability
# Downgrade protection to guarantee_incremental which uses zfs bookmarks instead of zfs holds.
# Thus, when we yank out the backup drive during replication
# - we might not be able to resume the interrupted replication step because the partially received `to` snapshot of a `from`->`to` step may be pruned any time
# - but in exchange we get back the disk space allocated by `to` when we prune it
# - and because we still have the bookmarks created by `guarantee_incremental`, we can still do incremental replication of `from`->`to2` in the future
incremental: guarantee_incremental
snapshotting:
type: manual
pruning:
# no-op prune rule on sender (keep all snapshots), job `snapshot` takes care of this
keep_sender:
- type: regex
regex: ".*"
# retain
keep_receiver:
# longer retention on the backup drive, we have more space there
- type: grid
grid: 1x1h(keep=all) | 24x1h | 360x1d
regex: "^zrepl_.*"
# retain all non-zrepl snapshots on the backup drive
- type: regex
negate: true
regex: "^zrepl_.*"
# This job receives from job `push_to_drive` into `backuppool/zrepl/sink/myhostname`
- type: sink
name: backuppool_sink
root_fs: "zfsbackup/zrepl"
serve:
type: local
listener_name: backuppool_sink
-
just enable the zrepl service
-
Sometimes I check zrepl status & force start the backup job with shift s if the job has errored.
-
You may want to adjust the number of snapshots that are taken (I’ve never needed them & am going to start taking them every 4 hours)
-
If you run LXD you may want to exclude some of those datasets (I just noticed they’ve screwed up my backups for a 2nd time)
Following the “ABC of backups” - I also use Vorta Backup to send backups to Borg Backup on remote storage.
Ok I’m trying. This is my zrepl.yml:
marco@tsaroo ~ $ cat /etc/zrepl/zrepl.yml
# This config serves as an example for a local zrepl installation that
# backups the entire zpool `system` to `backuppool/zrepl/sink`
#
# The requirements covered by this setup are described in the zrepl documentation's
# quick start section which inlines this example.
#
# CUSTOMIZATIONS YOU WILL LIKELY WANT TO APPLY:
# - adjust the name of the production pool `system` in the `filesystems` filter of jobs `snapjob` and `push_to_drive`
# - adjust the name of the backup pool `backuppool` in the `backuppool_sink` job
# - adjust the occurences of `myhostname` to the name of the system you are backing up (cannot be easily changed once you start replicating)
# - make sure the `zrepl_` prefix is not being used by any other zfs tools you might have installed (it likely isn't)
global:
logging:
- type: "stdout"
level: "warn"
format: "human"
- type: "syslog"
level: "warn"
format: "human"
jobs:
# this job takes care of snapshot creation + pruning
- name: snapjob
type: snap
filesystems: {
"lpool/root/gentoo-kde-systemd": true,
"lpool/home/gentoo-kde-systemd": true,
}
# create snapshots with prefix `zrepl_` every 15 minutes
snapshotting:
type: periodic
interval: 15m
prefix: zrepl_
pruning:
keep:
# fade-out scheme for snapshots starting with `zrepl_`
# - keep all created in the last hour
# - then destroy snapshots such that we keep 24 each 1 hour apart
# - then destroy snapshots such that we keep 14 each 1 day apart
# - then destroy all older snapshots
- type: grid
grid: 1x1h(keep=all) | 24x1h | 14x1d
regex: "^zrepl_.*"
# keep all snapshots that don't have the `zrepl_` prefix
- type: regex
negate: true
regex: "^zrepl_.*"
# This job pushes to the local sink defined in job `backuppool_sink`.
# We trigger replication manually from the command line / udev rules using
# `zrepl signal wakeup push_to_drive`
- type: push
name: push_to_drive
connect:
type: local
listener_name: backuppool_sink
client_identity: tsaroo
filesystems: {
"lpool/root/gentoo-kde-systemd": true
#"lpool/home/gentoo-kde-systemd": true
}
send:
encrypted: true
replication:
protection:
initial: guarantee_resumability
# Downgrade protection to guarantee_incremental which uses zfs bookmarks instead of zfs holds.
# Thus, when we yank out the backup drive during replication
# - we might not be able to resume the interrupted replication step because the partially received `to` snapshot of a `from`->`to` step may be pruned any time
# - but in exchange we get back the disk space allocated by `to` when we prune it
# - and because we still have the bookmarks created by `guarantee_incremental`, we can still do incremental replication of `from`->`to2` in the future
incremental: guarantee_incremental
snapshotting:
type: manual
pruning:
# no-op prune rule on sender (keep all snapshots), job `snapshot` takes care of this
keep_sender:
- type: regex
regex: ".*"
# retain
keep_receiver:
# longer retention on the backup drive, we have more space there
- type: grid
grid: 1x1h(keep=all) | 24x1h | 360x1d
regex: "^zrepl_.*"
# retain all non-zrepl snapshots on the backup drive
- type: regex
negate: true
regex: "^zrepl_.*"
# This job receives from job `push_to_drive` into `backuppool/zrepl/sink/myhostname`
- type: sink
name: backup
root_fs: "zroot/linux/tsaroo"
serve:
type: local
listener_name: backuppool_sink
when I launch zrepl status I get this:
╔═════════════════════════════════════════════╗┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
║jobs ║│Job: push_to_drive │
║├──backup ║│Type: push │
║├──push_to_drive ║│ │
║└──snapjob ║│Replication: │
║ ║│ Attempt #1 │
║ ║│ Status: filesystem-error │
║ ║│ Last Run: 2025-03-15 10:58:05 +0100 CET (lasted 0s) │
║ ║│ Problem: one or more of the filesystems encountered errors │
║ ║│ Progress: [==================================================>] 0 B / 0 B @ 0 B/s │
║ ║│ lpool/root/gentoo-kde-systemd PLANNING-ERROR (step 0/0, 0 B/0 B) validate send arguments: encrypted send mandated by policy, but filesystem "lpool/root/gentoo-kde-systemd" is not encrypted │
║ ║│ │
║ ║│Pruning Sender: │
║ ║│ Status: ExecErr │
║ ║│ Progress: [>] 0/0 snapshots │
║ ║│ lpool/root/gentoo-kde-systemd ERROR: replication cursor bookmark does not exist (one successful replication is required before pruning works) │
║ ║│ │
║ ║│ │
║ ║│Pruning Receiver: │
║ ║│ Status: Done │
║ ║│ nothing to do │
║ ║│ │
║ ║│Snapshotting: │
║ ║│ Type: manual │
║ ║│ <no details available> │
║ ║│
I don’t understand what I wrong. I’m trying to use my second nvme, with freebsd, how disk backup
If i comment line encrypted I get this:
╔═════════════════════════════════════════════╗┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
║jobs ║│Job: push_to_drive │
║├──backup ║│Type: push │
║├──push_to_drive ║│ │
║└──snapjob ║│Replication: │
║ ║│ Attempt #1 │
║ ║│ Status: filesystem-error │
║ ║│ Last Run: 2025-03-15 11:01:03 +0100 CET (lasted 0s) │
║ ║│ Problem: one or more of the filesystems encountered errors │
║ ║│ Progress: [>--------------------------------------------------] 7.0 MiB / 44.4 GiB @ 0 B/s │
║ ║│ lpool/root/gentoo-kde-systemd STEP-ERROR (step 1/1, 7.0 MiB/44.4 GiB) server error: cannot create placeholder filesystem zroot/linux/tsaroo/tsaroo/lpool: placeholder filesystem encryption handling is unspecified in receiver config│
║ ║│ │
║ ║│Pruning Sender: │
║ ║│ Status: ExecErr │
║ ║│ Progress: [>] 0/0 snapshots │
║ ║│ lpool/root/gentoo-kde-systemd ERROR: replication cursor bookmark does not exist (one successful replication is required before pruning works) │
║ ║│ │
║ ║│ │
║ ║│Pruning Receiver: │
║ ║│ Status: Done │
║ ║│ nothing to do │
║ ║│ │
║ ║│Snapshotting: │
║ ║│ Type: manual │
║ ║│ <no details available> │
║ ║│
seems to be a bug - try:
encryption: inherit
& if this does not work - do the first replication with off (as you have been doing)
after spending several hours releasing a LOT of zfs holds & destroying all the snapshots - I’m leaning towards trying sanoid - it looks simpler to configure & will probably be less error prone (my LXD datasets have messed things up twice now with zrepl)
I think with any zfs snapshot app - you need to check your zfs holds & exclude datasets held by any other app (like LXD).
Thanks so much, I’m solved first step, now I would like to try to automized this work
in Linux there is a zrepl systemd service I enable - maybe the FreeBSD maintainers added something similar ? (or just write a script that runs zrepl daemon)
Yes, I’m using systemd service on linux and snapshots are automatic, but for send/recv options I must use zrepl status with manual operation
and if is possible I would like to create backup on another external drive. So I have snapshot on my pool and two backup on anothers to disks
Maybe I solved with automatization of send/recv
I think the zrepl service runs whatever jobs are configured in /etc/zrepl/zrepl.yml (it does on my system including the file sync)
