Adding special mirror with different disk than pool's disks

OpenZFS
1

Hello,

i have a ZFS pool with:

6x 4TB SSD

	Logical  Sector size:                   512 bytes [ Supported: 512 ]
	Physical Sector size:                  4096 bytes

For now, everything works like a charm.

I want to add special mirror for metadata storage to speedup the pool ( i have ~50M of files ),

so, i want to re-use 500GB SSD disk i have.

Logical  Sector size:                   512 bytes [ Supported: 512 ]
Physical Sector size:                  4096 bytes

However, i have

[08:36:58] [root@server ~]# zpool add   backups special mirror /dev/sdb4 /dev/sde4
cannot add to 'backups': adding devices with different physical sector sizes is not allowed

My current pool

[08:37:16] [root@server ~]# zdb -C backups | grep -E 'ashift|vdev_type|path'
                ashift: 9

Do you have an idea ?

I tried with ashift=9 and ashift=12, but i hve the same error.

Thanks !

2

I have 128GB ram, so i configured this

options zfs zfs_arc_max=103079215104
options zfs zfs_arc_min=85899345920
options zfs zfs_arc_meta_limit=85899345920
options zfs zfs_arc_meta_min=64424509440

It should permit to store inodes, files list etc.. in the ARC ram, i will see if it’s enought for all.

3

You are going to destroy your SSDs due to write amplification if you keep the pool like that. ashift=12 is mandatory with them because their actual sector size is 4K (2^12).

Move the data off them, destroy the pool, recreate with ashift=12.

Then try adding your special mirror again (with ashift=12 as well).

4

I tried to add 4Kn (rust) drives to a pool with ashift=9 vdevs once. Everything ground to a halt, it was quite stressful, and I will never do that again.

This may not apply to your situation but frequently with NVMe in particular you can reformat them to different logical sector sizes and it might be worth checking before you fill them up with data again.

5

If the drive reports 4096 capability then 512 is only offered for compatibility and the write amplification will occur (and frankly effectively every SSD will have a larger page size internally anyway).

If there is an SSD involved anywhere, ashift=12, period. And frankly it should be the minimum these days because even with old spinners with native 512-byte sectors a 4K ashift has very little downside but allows upgrading the devices without killing them.

6

In my experience most NVMe drives offer higher throughput and lower latency at ashift=9 than at ashift=12. Which surprised the hell out of me as I was discovering it, but nevertheless, that’s typically the outcome of careful testing.

I suspect this is at least in part due to the massive parallelism possible with NVMe. AHCI (underlying SATA) only allows for a single command queue 32 operations deep, whereas NVMe supports 65,535 queues 65,536 operations deep each.

It’s a mistake to think NVMe drives are automatically superior to SATA drives, but it’s equally a mistake to think the two technologies behave in the same way when you get anywhere close to the bare metal.