I have a raidz1 pool with 5, 4TB drives. I am in the process of replacing the drives with 8TB drives.
My question is, if using the replace command (both the old and new drive are still connected and in good condition) can I run replace on more that one drive at a time?
Example, start replacing drive 1A with drive 1B, once it starts and is still resilvering, run the replace command again to replace drive 2A with drive 2B?
If this can be done would I still have redundancy in case of a failure along the way?
So resilvering multiple drives at the same time for replacement is possible. I will ask you though, you’re entering more dangerous territory, e.g. data loss possibility during a rebuild with larger drives and the use of raidz1.
How important is your data? Is it backed up elsewhere? Do you have the drive bays to set up a new pool for the 8TB drives in a raidz2 configuration?
If you went to a 4+2 raidz2 configuration you could still double your capacity, but your data would be much safer from drive failure. Currently you have ~14.5 TiB (4 data drives @ 3.638 TiB each) of usable capacity. Even staying at 5 total drives would bump you to ~21.8 TiB (7.276 TiB x 3) in a raidz2 setup.
So on to your question:
I’m assuming that your pool is at the feature level of v0.8.6+. If not, then I’m not sure what happens. From the answer to the same question elsewhere:
zfs prioritizes resilvering the first to-be-resilvered disk, if/when additional disks are added to the resilvering queue. This is a default behaviour, it is a feature flag called resilver_defer: https://openzfs.github.io/openzfs-docs/man/7/zpool-features.7.html#resilver_defer
Resilvers can’t be paused and resumed, therefore adding more to-be-resilvered disks would force the resilver to restart on the first disk as well. The way resilvering works prioritizes the restoration of redundancy over being able to pause/resume resilvering.
The reason for this feature is that it’s the safest way to handle any scenario. Imagine a disk dies and you replace it. You want that resilver to complete asap to restore full redundancy. But at 90% completion another disk fails and that resilver to a spare is queued. Without that deferring feature zfs would restart that 90% completed resilver and resilver both disks at once from the beginning again, thus delaying restoring redundancy.
You can override this behavior by issuing a command right after running the replace command on all the drives in order to abort the current prioritized resilver and start over with resilvering all disks at once:
zpool resilver <pool>