Replacement strategy for old disks in mirrored setup

OpenZFS
1

Hi all!

Another reddit migrant seeking your wisdom here :slight_smile:

I have a pool consisting of 2 mirrors with 2 disks ea. Now I want to replace the disks from one of the mirrors as they are, well, ancient.

mirror-0 has two 3TB drives that should go, mirror-1 has two 6TB drives I would like to replace sometime next year probably. I only have those 4 bays to “properly” install the drives.

I’m wondering what the best way to do this would be:

  1. take the disks from the mirror-1 out of the server, so I can attach the two new disks to mirror-0. Let them resilver, remove the two old 3TB disks(, grow FS) and then insert the disks from mirror-1 again

  2. remove one 3TB disk from the mirror-0, install/attach new drive in its place, resilver, remove second 3TB drive, install/attach second new drive, resilver again.

  3. Attach a new disk via USB3 and resilver, remove both old drives from pool, install both new drives as SATA, attach second new drive to mirror and resilver.

Option 1 would of course degrade the pool until the resilvering is done and make the system barely usable. However it might be faster(?) and safer(?) to be able to read from both old drives in mirror-0 for the resilvering?

Option 2 would keep the system available (I would try to write as little as possible though), but I guess it would take longer to resilver (as it’s sequential) and puts initial stress on one of the old drives to get the first new drive resilvered.

Option 3: I have no clue if the “use a USB to SATA connector to do the initial resilver” is a brilliant or terrible idea. Would keep the whole pool available, stress on the old drives low and would “just” cot me the time to resilver twice.

Am I missing an obvious fourth option?

Thanks all!

2

You can do the USB3 thing if you like, though it’s generally not worth it. Essentially, you resilver in on USB3, then remove one of the disks in your mirror, then you export the pool, move the USB3-d disk into the main system, then reimport the pool using -d /dev/disk/by-id so it will find the new one.

This usually isn’t worth it, though. Instead, make sure you’ve got a recently-completed scrub and no errors, then use zpool split to turn your pool of mirror vdevs into two identical pools of single-disk vdevs. Then zpool export the split pool and remove those disks–that’s your backup, if things go horribly wrong.

https://openzfs.github.io/openzfs-docs/man/master/8/zpool-split.8.html

Now that you’ve got the empty bays available, pop in two of your new disks, zpool attach them in order to upgrade your pool back to mirror vdevs, and let them resilver in; then, once they’ve resilvered in, zpool offline and zpool remove the remaining originals, physically replace them with the remaining new drives, zpool attach those, and Bob’s your uncle.