Can't add SLOG mirror, sector size issue

Proxmox
1

Hi all,

I have a 3 drive raidz pool with 3 8TB drives. I’m trying to add 2 Optane NVMe drives for slog, but I’m getting this error:

cannot add to 'tank': adding devices with different physical sector sizes is not allowed

I have another proxmox system that’s functionally identical, 4 spinners in raidz, with 2 optane drives for slog. Only difference I can see (including the sector sizes on both systems) is that the working system is running zfs-kmod-2.2.0-pve1 and the new one with the issue is running zfs-kmod-2.2.4-pve1.

The spinners in both systems report these sector sizes Sector size (logical/physical): 512 bytes / 4096 bytes and the optane nvme drives report these sizes Sector size (logical/physical): 512 bytes / 512 bytes

Seems very odd to me. Anyone spot something I’ve missed?

2

Ensure the nvme-cli package is installed and run:
nvme id-ns /dev/nvme{0…2}n1 to see available namespaces (lbaf, 512/4k) at the end of each one.
You can reformat to 4k sectors if supported by your model and after that try again to add as slog.

1 Like
3

I have a similar problem with an ordinary SATA SSD - it reports as 512/512 with parted print - is there any way to change it to 4096 logical and use it for a LOG vdev or is it just too old? I went and wiped the SSD with GParted and then tried doing zfs add pool log device; ZFS creates the partitions but then OpenZFS rejects it:

cannot add to 'pool': adding devices with different physical sector sizes is not allowed

I am using another SATA SSD for cache device - this also reports 512/512 but is accepted as a cache device?

BTW I think this thread ought to be in OpenZFS rather than Proxmox :slight_smile:

4

Have you tried it with -o ashift=12? Not sure if this would help?

2 Likes
5

zpool add poolname log devicepath -o ashift=12 should work just fine.

1 Like
6

You are quite right! it did the trick!

What puzzles me is that it was not needed when adding a cache device - did OpenZFS automagically determine the sector/ashift size or is it now at a non-optimal setting? And why is there this inconsistency?

7

It depends on what ZFS detects as the native sector size for the disk, as to what it will set it to. For example:

> sudo parted /dev/nvme0n1
[sudo] password for jtuckey: 
GNU Parted 3.6
Using /dev/nvme0n1
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p                                                                
Model: CT2000P3PSSD8 (nvme)
Disk /dev/nvme0n1: 2000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

This disk reports a 512B sector size, so it will try to set ashift=9 as 2^9 = 512

This disk:

remoteconsole@VM:~/remote-console-access/ansible$ sudo parted /dev/sda
GNU Parted 3.4
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p                                                                
Model: Msft Virtual Disk (scsi)
Disk /dev/sda: 32.2GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Reports a physical sector size of 4096B, so it will try to set ashift=12 as 2^12 = 4096.

8

All spinning rust in this pool are 512/4096. All three of SSDs in this box - the NVMe boot disk, the cache and now the log disk - are both 512/512. That is why I am puzzled - when adding the cache disk I was not warned/prompted that there was a sector size mismatch - only when adding the log.

Does this mean that my cache in ZFS terms is ashift=9 when the rest of the pool is ashift=12 (log device was ‘forced’ to ashift=12 when adding it)?

9

Oh right, I see. Yeah, I would guess that the log device needs to match the rest of the storage, while the cache can be mismatched.

You could try this command to see the ashift sizes: zpool get ashift zroot all-vdevs

Be aware there’s a display bug that makes the output a little confusing, discussed here:

10

mine says:

# zpool get ashift fatagnus all-vdevs
NAME      PROPERTY  VALUE   SOURCE
root-0    ashift    0       -
cannot get vdev property name from ata-PNY_CS900_480GB_SSD_PNY31190023700100454 in fatagnus: invalid argument for this pool operation
cannot get vdev property ashift from ata-PNY_CS900_480GB_SSD_PNY31190023700100454 in fatagnus: invalid argument for this pool operation
raidz2-0  ashift    12      -
ata-ST4000LM024-2AN17V_WFF0HR7Q  ashift    9       -
ata-ST8000DM004-2ZD188_WSC19QM2  ashift    9       -
ata-TOSHIBA_HDWD130_Z053RNKAS  ashift    9       -
ata-TOSHIBA_HDWD130_Z053RR2AS  ashift    9       -
ata-WDC_WD40EFZX-68AWUN0_WD-WX22DB1DF7PL  ashift    9       -
ata-WDC_WD40EFZX-68AWUN0_WD-WX52DB1ASC33  ashift    9       -
ata-SAMSUNG_MZ7LF120HCHP-000L1_S25ZNXAGA10897  ashift    12      -

Interestingly the device it cannot get the ashift value for is the cache device - maybe this explains why it did not complain about sector size?

1 Like
11

Yeah, that’s a weird one, not sure why it can’t show the ashift for the cache device.
You can see, however, that the raidz2 pool is set to 12, and that’s why the log device needs to be 12 also, so that sector sizes match.