Resilver speed slowing down significantly, concerned about losing pool

Function · December 6, 2025, 1:47pm

Five days ago I saw that one of my 8 drives in my RAID-Z2 is “faulted”, so I replaced it and ZFS started to resilver.

It started with 1G/s and an ETA of ~1 day. Next day it dropped to 600M/s and adjusted ETA. That next day I saw that a second drive is “faulted”, so another failed drive will make me lose the pool. (I have backups but restore will be painful).

This continued until today where it sits at 85% (same as yesterday) at 250M/s with an ETA of 19h (yesterday it was 15h).

When the resilver started, I stopped most services that access datasets on the pool. A few days ago, I stopped more services and unmounted all datasets so that the server is basically idle aside from the disk utilization. I also checked SMART and drive temps are between 30 and 45°C.

Since the speed is slowing down so much and it’s not even doing 1% per day anymore, I feel like it will take weeks to finish, maybe even until January.

Is there anything I can do to speed up the resilver or take other measures to prevent my pool from dying?

Edit: Current zpool status tank:

  pool: tank
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Mon Dec  1 20:48:19 2025
        97.6T / 114T scanned at 250M/s, 97.6T / 114T issued at 250M/s
        11.9T resilvered, 85.32% done, 19:33:50 to go
config:

        NAME                                         STATE     READ WRITE CKSUM
        tank                                         DEGRADED     0     0     0
          raidz2-0                                   DEGRADED     0     0     0
            scsi-SATA_TOSHIBA_MG09ACA1_X3M0A017FJDH  ONLINE       0     0     0
            replacing-1                              DEGRADED     0     0   341
              18371118026138877845                   UNAVAIL      0     0     0  was /dev/disk/by-id/scsi-SATA_TOSHIBA_MG09ACA1_X3M0A05KFJDH-part1
              scsi-35000039e18da5716                 ONLINE       0     0     0  (resilvering)
            scsi-SATA_TOSHIBA_MG09ACA1_X3M0A007FJDH  ONLINE       0     0     0
            scsi-SATA_TOSHIBA_MG09ACA1_X3M0A01CFJDH  FAULTED     46     0     0  too many errors
            scsi-SATA_TOSHIBA_MG09ACA1_53G0A0KYFJDH  ONLINE       0     0     0
            scsi-SATA_TOSHIBA_MG09ACA1_X3M0A011FJDH  ONLINE       0     0     0
            scsi-SATA_TOSHIBA_MG09ACA1_X3M0A00CFJDH  ONLINE       0     0     0
            scsi-SATA_TOSHIBA_MG09ACA1_X3M0A021FJDH  ONLINE       0     0     0

errors: No known data errors

deviantintegral · December 6, 2025, 3:03pm

Are you absolutely sure the drives are dying? Any chance you have a bad or loose cable, or an issue with whatever the drives are plugged into? Or what if the problem is writing to the new drive, not reading from the old ones?

It might be worth checking iostat.

Do you see errors in smartctl or dmesg?

HankB · December 6, 2025, 3:23pm

Do you know if the version of ZFS you’re using does sequential resilvering? https://openzfs.org/wiki/Scrub/Resilver_Performance states

… it is common to see resilver operations taking days to weeks.

I don’t know offhand what version of OpenZFS implemented sequential resilver, and I presume you’re using a recent enough version.

Do you know why the status shows

            replacing-1                              DEGRADED     0     0   341
...
            scsi-SATA_TOSHIBA_MG09ACA1_X3M0A01CFJDH  FAULTED     46     0     0  too many errors

Does that indicate problems with two drives or do those both refer to the same drive?

Function · December 6, 2025, 3:25pm

Are you absolutely sure the drives are dying?

The replaced and faulted ones, yes. The others don’t show errors yet but have similar spin-up-times and power-on-hours, so it’s not unlikely that some of the others are starting to fail too.

Selected SMART values for all drives

# for d in /dev/sd?; do echo "=== $d ==="; sudo smartctl -A "$d" | awk '/^(ID|  5|197|198|  3|  9)/ {printf "%-25s %-10s %-10s\n", $2, $10, $4}'; echo; done

=== /dev/sda ===
ATTRIBUTE_NAME            RAW_VALUE  VALUE
Spin_Up_Time              8842       100
Reallocated_Sector_Ct     0          100
Power_On_Hours            15944      061
Current_Pending_Sector    0          100
Offline_Uncorrectable     0          100

=== /dev/sdb ===
ATTRIBUTE_NAME            RAW_VALUE  VALUE
Spin_Up_Time              9112       100
Reallocated_Sector_Ct     0          100
Power_On_Hours            115        100
Current_Pending_Sector    0          100
Offline_Uncorrectable     0          100

=== /dev/sdc ===
ATTRIBUTE_NAME            RAW_VALUE  VALUE
Spin_Up_Time              8846       100
Reallocated_Sector_Ct     0          100
Power_On_Hours            15944      061
Current_Pending_Sector    0          100
Offline_Uncorrectable     0          100

=== /dev/sdd ===
ATTRIBUTE_NAME            RAW_VALUE  VALUE
Spin_Up_Time              8827       100
Reallocated_Sector_Ct     2          100
Power_On_Hours            15944      061
Current_Pending_Sector    9          100
Offline_Uncorrectable     10         100

=== /dev/sde ===
ATTRIBUTE_NAME            RAW_VALUE  VALUE
Spin_Up_Time              8960       100
Reallocated_Sector_Ct     0          100
Power_On_Hours            15944      061
Current_Pending_Sector    0          100
Offline_Uncorrectable     0          100

=== /dev/sdf ===
ATTRIBUTE_NAME            RAW_VALUE  VALUE
Spin_Up_Time              8873       100
Reallocated_Sector_Ct     0          100
Power_On_Hours            15944      061
Current_Pending_Sector    0          100
Offline_Uncorrectable     0          100

=== /dev/sdg ===
ATTRIBUTE_NAME            RAW_VALUE  VALUE
Spin_Up_Time              8732       100
Reallocated_Sector_Ct     0          100
Power_On_Hours            15944      061
Current_Pending_Sector    0          100
Offline_Uncorrectable     0          100

=== /dev/sdh ===
ATTRIBUTE_NAME            RAW_VALUE  VALUE
Spin_Up_Time              8697       100
Reallocated_Sector_Ct     0          100
Power_On_Hours            15944      061
Current_Pending_Sector    0          100
Offline_Uncorrectable     0          100

=== /dev/sdi ===
ATTRIBUTE_NAME            RAW_VALUE  VALUE
Reallocated_Sector_Ct     0          100
Power_On_Hours            15950      100
Current_Pending_Sector    0          100
Offline_Uncorrectable     0          100

=== /dev/sdj ===
ATTRIBUTE_NAME            RAW_VALUE  VALUE
Reallocated_Sector_Ct     0          100
Power_On_Hours            15965      100
Current_Pending_Sector    0          100
Offline_Uncorrectable     0          100

Any chance you have a bad or loose cable, or an issue with whatever the drives are plugged into?

I checked all cables and the HBA they’re attached to. Everything is seated properly.

Or what if the problem is writing to the new drive, not reading from the old ones?

I don’t know how I could tell, but maybe that’s the issue.

It might be worth checking iostat.

I ran iostat -dx /dev/sd? and it looks like this almost all the time. I’m an amateur but i’d interpret this as the other drives being the bottleneck, not the new drive:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.04    0.00    0.50    8.40    0.00   91.06

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda            322.50   1604.00     0.00   0.00    9.32     4.97    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    3.01  99.55
sdb              0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
sdc            196.50    938.00     0.00   0.00   15.21     4.77    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.99  99.45
sdd              0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
sde            210.50    938.00     0.00   0.00   13.74     4.46    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.89  99.15
sdf            308.50   1482.00     0.00   0.00    9.71     4.80    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    3.00  99.20
sdg            309.00   1556.00     0.00   0.00    9.66     5.04    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.98  99.05
sdh            307.50   1682.00     0.00   0.00    9.76     5.47    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    3.00  99.35
sdi              0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
sdj              0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00

Do you see errors in smartctl or dmesg?

For smartctl, see first section.

For dmesg, I threw some keywords into a grep filter and this is what I got, but I really don’t know how to interpret this:

dmesg

# dmesg --human | grep -iE 'error|fail|warn|ata|ahci|sd|scsi|i/o'
[  +0.000000] BIOS-e820: [mem 0x00000000bcd65000-0x00000000bcda0fff] ACPI data
[  +0.000000] reserve setup_data: [mem 0x0000000000000000-0x000000000009ffff] usable
[  +0.000000] reserve setup_data: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[  +0.000000] reserve setup_data: [mem 0x0000000000100000-0x0000000009bfefff] usable
[  +0.000000] reserve setup_data: [mem 0x0000000009bff000-0x0000000009ffffff] reserved
[  +0.000000] reserve setup_data: [mem 0x000000000a000000-0x000000000a1fffff] usable
[  +0.000000] reserve setup_data: [mem 0x000000000a200000-0x000000000a20dfff] ACPI NVS
[  +0.000000] reserve setup_data: [mem 0x000000000a20e000-0x000000000affffff] usable
[  +0.000000] reserve setup_data: [mem 0x000000000b000000-0x000000000b01ffff] reserved
[  +0.000000] reserve setup_data: [mem 0x000000000b020000-0x00000000b75f6017] usable
[  +0.000000] reserve setup_data: [mem 0x00000000b75f6018-0x00000000b76b8057] usable
[  +0.000000] reserve setup_data: [mem 0x00000000b76b8058-0x00000000b76b9017] usable
[  +0.000000] reserve setup_data: [mem 0x00000000b76b9018-0x00000000b76c5a57] usable
[  +0.000000] reserve setup_data: [mem 0x00000000b76c5a58-0x00000000bb623fff] usable
[  +0.000000] reserve setup_data: [mem 0x00000000bb624000-0x00000000bcd64fff] reserved
[  +0.000000] reserve setup_data: [mem 0x00000000bcd65000-0x00000000bcda0fff] ACPI data
[  +0.000000] reserve setup_data: [mem 0x00000000bcda1000-0x00000000bd47bfff] ACPI NVS
[  +0.000000] reserve setup_data: [mem 0x00000000bd47c000-0x00000000bdffefff] reserved
[  +0.000000] reserve setup_data: [mem 0x00000000bdfff000-0x00000000beffffff] usable
[  +0.000000] reserve setup_data: [mem 0x00000000bf000000-0x00000000bfffffff] reserved
[  +0.000000] reserve setup_data: [mem 0x00000000f0000000-0x00000000f7ffffff] reserved
[  +0.000000] reserve setup_data: [mem 0x00000000fd100000-0x00000000fd1fffff] reserved
[  +0.000000] reserve setup_data: [mem 0x00000000fd500000-0x00000000fd6fffff] reserved
[  +0.000000] reserve setup_data: [mem 0x00000000fea00000-0x00000000fea0ffff] reserved
[  +0.000000] reserve setup_data: [mem 0x00000000feb80000-0x00000000fec01fff] reserved
[  +0.000000] reserve setup_data: [mem 0x00000000fec10000-0x00000000fec10fff] reserved
[  +0.000000] reserve setup_data: [mem 0x00000000fec30000-0x00000000fec30fff] reserved
[  +0.000000] reserve setup_data: [mem 0x00000000fed00000-0x00000000fed00fff] reserved
[  +0.000000] reserve setup_data: [mem 0x00000000fed40000-0x00000000fed44fff] reserved
[  +0.000000] reserve setup_data: [mem 0x00000000fed80000-0x00000000fed8ffff] reserved
[  +0.000000] reserve setup_data: [mem 0x00000000fedc2000-0x00000000fedcffff] reserved
[  +0.000000] reserve setup_data: [mem 0x00000000fedd4000-0x00000000fedd5fff] reserved
[  +0.000000] reserve setup_data: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
[  +0.000000] reserve setup_data: [mem 0x0000000100000000-0x000000083f37ffff] usable
[  +0.000000] reserve setup_data: [mem 0x000000083f380000-0x000000083fffffff] reserved
[  +0.000003] ACPI: RSDP 0x00000000BD465014 000024 (v02 ALASKA)
[  +0.000003] ACPI: XSDT 0x00000000BD464728 0000DC (v01 ALASKA A M I    01072009 AMI  01000013)
[  +0.000003] ACPI: DSDT 0x00000000BCD8B000 0067DE (v02 ALASKA A M I    01072009 INTL 20120913)
[  +0.000002] ACPI: SSDT 0x00000000BCD98000 008CE9 (v02 AMD    AmdTable 00000002 MSFT 04000000)
[  +0.000002] ACPI: SSDT 0x00000000BCD94000 003B45 (v02 AMD    AMD AOD  00000001 INTL 20120913)
[  +0.000002] ACPI: SSDT 0x00000000BCD93000 0001AD (v02 ALASKA CPUSSDT  01072009 AMI  01072009)
[  +0.000002] ACPI: SSDT 0x00000000BCD7F000 0030FB (v02 AMD    AmdTable 00000001 AMD  00000001)
[  +0.000002] ACPI: SSDT 0x00000000BCD7C000 00052C (v02 AMD    ArticNOI 00000001 INTL 20120913)
[  +0.000002] ACPI: SSDT 0x00000000BCD78000 0037C4 (v02 AMD    ArticN   00000001 INTL 20120913)
[  +0.000002] ACPI: SSDT 0x00000000BCD75000 00007D (v02 AMD    ArticDIS 00000001 INTL 20120913)
[  +0.000001] ACPI: SSDT 0x00000000BCD73000 0010AF (v02 AMD    ArticC   00000001 INTL 20120913)
[  +0.000002] ACPI: SSDT 0x00000000BCD72000 0000BF (v01 AMD    AmdTable 00001000 INTL 20120913)
[  +0.000001] ACPI: Reserving DSDT table memory at [mem 0xbcd8b000-0xbcd917dd]
[  +0.000001] ACPI: Reserving SSDT table memory at [mem 0xbcd98000-0xbcda0ce8]
[  +0.000000] ACPI: Reserving SSDT table memory at [mem 0xbcd94000-0xbcd97b44]
[  +0.000001] ACPI: Reserving SSDT table memory at [mem 0xbcd93000-0xbcd931ac]
[  +0.000000] ACPI: Reserving SSDT table memory at [mem 0xbcd7f000-0xbcd820fa]
[  +0.000001] ACPI: Reserving SSDT table memory at [mem 0xbcd7c000-0xbcd7c52b]
[  +0.000000] ACPI: Reserving SSDT table memory at [mem 0xbcd78000-0xbcd7b7c3]
[  +0.000001] ACPI: Reserving SSDT table memory at [mem 0xbcd75000-0xbcd7507c]
[  +0.000000] ACPI: Reserving SSDT table memory at [mem 0xbcd73000-0xbcd740ae]
[  +0.000001] ACPI: Reserving SSDT table memory at [mem 0xbcd72000-0xbcd720be]
[  +0.000004] NODE_DATA(0) allocated [mem 0x83f355000-0x83f37ffff]
[  +0.048044] Memory: 32619888K/33477716K available (22528K kernel code, 4441K rwdata, 14384K rodata, 4912K init, 4796K bss, 857568K reserved, 0K cma-reserved)
[  +0.000018] APIC: Switch to symmetric I/O mode setup
[  +0.005975] ACPI: _OSC evaluation for CPUs failed, trying _PDC
[  +0.000000] SCSI subsystem initialized
[  +0.000560] libata version 3.00 loaded.
[  +0.000017] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 243)
[  +0.020925] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[  +0.017521] RAS: Correctable Errors collector initialized.
[  +0.000092] Write protecting the kernel read-only data: 38912k
[  +0.000542] Freeing unused kernel image (rodata/data gap) memory: 2000K
[  +0.001880] ahci 0000:02:00.1: version 3.0
[  +0.003744] ahci 0000:02:00.1: SSS flag set, parallel bus scan disabled
[  +0.000194] ahci 0000:02:00.1: AHCI 0001.0301 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
[  +0.000244] ahci 0000:02:00.1: flags: 64bit ncq sntf stag pm led clo only pmp pio slum part sxs deso sadm sds apst
[  +0.001512] scsi host0: ahci
[  +0.000388] scsi host1: ahci
[  +0.000311] scsi host2: ahci
[  +0.000302] scsi host3: ahci
[  +0.000326] scsi host4: ahci
[  +0.000296] scsi host5: ahci
[  +0.000239] ata1: SATA max UDMA/133 abar m131072@0xfbe80000 port 0xfbe80100 irq 42 lpm-pol 0
[  +0.000338] ata2: SATA max UDMA/133 abar m131072@0xfbe80000 port 0xfbe80180 irq 42 lpm-pol 0
[  +0.000348] ata3: SATA max UDMA/133 abar m131072@0xfbe80000 port 0xfbe80200 irq 42 lpm-pol 0
[  +0.000358] ata4: SATA max UDMA/133 abar m131072@0xfbe80000 port 0xfbe80280 irq 42 lpm-pol 0
[  +0.000370] ata5: SATA max UDMA/133 abar m131072@0xfbe80000 port 0xfbe80300 irq 42 lpm-pol 0
[  +0.000382] ata6: SATA max UDMA/133 abar m131072@0xfbe80000 port 0xfbe80380 irq 42 lpm-pol 0
[  +0.058674] mpt2sas_cm0: overriding NVDATA EEDPTagMode setting
[  +0.000585] scsi host6: Fusion MPT SAS Host
[  +0.066257] ata1: SATA link down (SStatus 0 SControl 300)
[  +0.049246] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[  +0.002154] ata2.00: ATA-9: Intenso SSD Sata III, W0406A0, max UDMA/133
[  +0.001001] ata2.00: 1000215216 sectors, multi 1: LBA48 NCQ (depth 32), AA
[  +0.004146] ata2.00: configured for UDMA/133
[  +0.000333] scsi: waiting for bus probes to complete ...
[  +0.501773] scsi 6:0:0:0: Direct-Access     ATA      TOSHIBA MG09ACA1 0105 PQ: 0 ANSI: 6
[  +0.000641] scsi 6:0:0:0: SATA: handle(0x0009), sas_addr(0x4433221100000000), phy(0), device_name(0x0000000000000000)
[  +0.000568] scsi 6:0:0:0: enclosure logical id (0x500605b004d48020), slot(3)
[  +0.000422] scsi 6:0:0:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[  +0.000591] scsi 6:0:0:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
[  +0.175793] scsi 6:0:1:0: Direct-Access     ATA      TOSHIBA MG09ACA1 0107 PQ: 0 ANSI: 6
[  +0.000662] scsi 6:0:1:0: SATA: handle(0x000a), sas_addr(0x4433221101000000), phy(1), device_name(0x0000000000000000)
[  +0.000601] scsi 6:0:1:0: enclosure logical id (0x500605b004d48020), slot(2)
[  +0.000437] scsi 6:0:1:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[  +0.000617] scsi 6:0:1:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
[  +0.175346] scsi 6:0:2:0: Direct-Access     ATA      TOSHIBA MG09ACA1 0105 PQ: 0 ANSI: 6
[  +0.000684] scsi 6:0:2:0: SATA: handle(0x000b), sas_addr(0x4433221102000000), phy(2), device_name(0x0000000000000000)
[  +0.000606] scsi 6:0:2:0: enclosure logical id (0x500605b004d48020), slot(1)
[  +0.000427] scsi 6:0:2:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[  +0.000631] scsi 6:0:2:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
[  +0.175772] scsi 6:0:3:0: Direct-Access     ATA      TOSHIBA MG09ACA1 0105 PQ: 0 ANSI: 6
[  +0.000668] scsi 6:0:3:0: SATA: handle(0x000c), sas_addr(0x4433221103000000), phy(3), device_name(0x0000000000000000)
[  +0.000604] scsi 6:0:3:0: enclosure logical id (0x500605b004d48020), slot(0)
[  +0.000451] scsi 6:0:3:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[  +0.000633] scsi 6:0:3:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
[  +0.177462] scsi 6:0:4:0: Direct-Access     ATA      TOSHIBA MG09ACA1 0105 PQ: 0 ANSI: 6
[  +0.000684] scsi 6:0:4:0: SATA: handle(0x000d), sas_addr(0x4433221104000000), phy(4), device_name(0x0000000000000000)
[  +0.000605] scsi 6:0:4:0: enclosure logical id (0x500605b004d48020), slot(7)
[  +0.000431] scsi 6:0:4:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[  +0.000632] scsi 6:0:4:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
[  +0.175135] scsi 6:0:5:0: Direct-Access     ATA      TOSHIBA MG09ACA1 0105 PQ: 0 ANSI: 6
[  +0.000688] scsi 6:0:5:0: SATA: handle(0x000e), sas_addr(0x4433221105000000), phy(5), device_name(0x0000000000000000)
[  +0.000610] scsi 6:0:5:0: enclosure logical id (0x500605b004d48020), slot(6)
[  +0.000442] scsi 6:0:5:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[  +0.000631] scsi 6:0:5:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
[  +0.176055] scsi 6:0:6:0: Direct-Access     ATA      TOSHIBA MG09ACA1 0105 PQ: 0 ANSI: 6
[  +0.000689] scsi 6:0:6:0: SATA: handle(0x000f), sas_addr(0x4433221106000000), phy(6), device_name(0x0000000000000000)
[  +0.000606] scsi 6:0:6:0: enclosure logical id (0x500605b004d48020), slot(5)
[  +0.000440] scsi 6:0:6:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[  +0.000630] scsi 6:0:6:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
[  +0.176782] scsi 6:0:7:0: Direct-Access     ATA      TOSHIBA MG09ACA1 0105 PQ: 0 ANSI: 6
[  +0.000670] scsi 6:0:7:0: SATA: handle(0x0010), sas_addr(0x4433221107000000), phy(7), device_name(0x0000000000000000)
[  +0.000604] scsi 6:0:7:0: enclosure logical id (0x500605b004d48020), slot(4)
[  +0.000448] scsi 6:0:7:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[  +0.000623] scsi 6:0:7:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
[  +0.000819] sd 6:0:0:0: Attached scsi generic sg0 type 0
[  +0.000119] sd 6:0:0:0: Power-on or device reset occurred
[  +0.000473] sd 6:0:1:0: Attached scsi generic sg1 type 0
[  +0.000108] sd 6:0:1:0: Power-on or device reset occurred
[  +0.000536] sd 6:0:2:0: Attached scsi generic sg2 type 0
[  +0.000117] sd 6:0:2:0: Power-on or device reset occurred
[  +0.000492] scsi 6:0:3:0: Attached scsi generic sg3 type 0
[  +0.000218] sd 6:0:3:0: Power-on or device reset occurred
[  +0.000320] scsi 6:0:4:0: Attached scsi generic sg4 type 0
[  +0.000262] sd 6:0:4:0: Power-on or device reset occurred
[  +0.000281] scsi 6:0:5:0: Attached scsi generic sg5 type 0
[  +0.000280] sd 6:0:5:0: Power-on or device reset occurred
[  +0.000267] scsi 6:0:6:0: Attached scsi generic sg6 type 0
[  +0.000236] sd 6:0:6:0: Power-on or device reset occurred
[  +0.000293] scsi 6:0:7:0: Attached scsi generic sg7 type 0
[  +0.000143] sd 6:0:7:0: Power-on or device reset occurred
[  +0.000259] scsi 1:0:0:0: Direct-Access     ATA      Intenso SSD Sata 6A0  PQ: 0 ANSI: 5
[  +0.000032] sd 6:0:0:0: [sda] 35156656128 512-byte logical blocks: (18.0 TB/16.4 TiB)
[  +0.000372] sd 6:0:1:0: [sdb] 35156656128 512-byte logical blocks: (18.0 TB/16.4 TiB)
[  +0.000198] sd 1:0:0:0: Attached scsi generic sg8 type 0
[  +0.000049] sd 1:0:0:0: [sdi] 1000215216 512-byte logical blocks: (512 GB/477 GiB)
[  +0.000007] sd 6:0:0:0: [sda] 4096-byte physical blocks
[  +0.000007] sd 6:0:2:0: [sdc] 35156656128 512-byte logical blocks: (18.0 TB/16.4 TiB)
[  +0.000001] sd 1:0:0:0: [sdi] Write Protect is off
[  +0.000002] sd 1:0:0:0: [sdi] Mode Sense: 00 3a 00 00
[  +0.000018] sd 1:0:0:0: [sdi] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[  +0.000017] sd 1:0:0:0: [sdi] Preferred minimum I/O size 512 bytes
[  +0.000281] sd 6:0:1:0: [sdb] 4096-byte physical blocks
[  +0.000132] sd 6:0:2:0: [sdc] 4096-byte physical blocks
[  +0.000009] sd 6:0:3:0: [sdd] 35156656128 512-byte logical blocks: (18.0 TB/16.4 TiB)
[  +0.000461] sd 6:0:4:0: [sde] 35156656128 512-byte logical blocks: (18.0 TB/16.4 TiB)
[  +0.000032] sd 6:0:3:0: [sdd] 4096-byte physical blocks
[  +0.000320]  sdi: sdi1 sdi9
[  +0.000051] sd 6:0:4:0: [sde] 4096-byte physical blocks
[  +0.000052] sd 1:0:0:0: [sdi] Attached SCSI disk
[  +0.000067] sd 6:0:5:0: [sdf] 35156656128 512-byte logical blocks: (18.0 TB/16.4 TiB)
[  +0.000004] sd 6:0:5:0: [sdf] 4096-byte physical blocks
[  +0.000514] sd 6:0:6:0: [sdg] 35156656128 512-byte logical blocks: (18.0 TB/16.4 TiB)
[  +0.000365] sd 6:0:7:0: [sdh] 35156656128 512-byte logical blocks: (18.0 TB/16.4 TiB)
[  +0.000240] sd 6:0:6:0: [sdg] 4096-byte physical blocks
[  +0.000337] sd 6:0:7:0: [sdh] 4096-byte physical blocks
[  +0.002404] sd 6:0:0:0: [sda] Write Protect is off
[  +0.000172] sd 6:0:0:0: [sda] Mode Sense: 7f 00 10 08
[  +0.000313] sd 6:0:2:0: [sdc] Write Protect is off
[  +0.000156] sd 6:0:2:0: [sdc] Mode Sense: 7f 00 10 08
[  +0.000085] sd 6:0:1:0: [sdb] Write Protect is off
[  +0.000202] sd 6:0:1:0: [sdb] Mode Sense: 7f 00 10 08
[  +0.000539] sd 6:0:5:0: [sdf] Write Protect is off
[  +0.000074] sd 6:0:3:0: [sdd] Write Protect is off
[  +0.000093] sd 6:0:5:0: [sdf] Mode Sense: 7f 00 10 08
[  +0.000152] sd 6:0:3:0: [sdd] Mode Sense: 7f 00 10 08
[  +0.000246] sd 6:0:4:0: [sde] Write Protect is off
[  +0.000151] sd 6:0:4:0: [sde] Mode Sense: 7f 00 10 08
[  +0.000866] sd 6:0:6:0: [sdg] Write Protect is off
[  +0.000148] sd 6:0:6:0: [sdg] Mode Sense: 7f 00 10 08
[  +0.001566] sd 6:0:7:0: [sdh] Write Protect is off
[  +0.000144] sd 6:0:7:0: [sdh] Mode Sense: 7f 00 10 08
[  +0.001059] sd 6:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
[  +0.000468] sd 6:0:2:0: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA
[  +0.000293] sd 6:0:1:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
[  +0.000859] sd 6:0:5:0: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA
[  +0.000028] sd 6:0:3:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA
[  +0.000358] sd 6:0:4:0: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA
[  +0.001022] sd 6:0:6:0: [sdg] Write cache: enabled, read cache: enabled, supports DPO and FUA
[  +0.001696] sd 6:0:7:0: [sdh] Write cache: enabled, read cache: enabled, supports DPO and FUA
[  +0.045521]  sda: sda1 sda9
[  +0.000469] sd 6:0:0:0: [sda] Attached SCSI disk
[  +0.013009]  sdd: sdd1 sdd9
[  +0.000049]  sdc: sdc1 sdc9
[  +0.000056]  sdb: sdb1 sdb9
[  +0.000212] sd 6:0:1:0: [sdb] Attached SCSI disk
[  +0.000389] sd 6:0:3:0: [sdd] Attached SCSI disk
[  +0.000096] sd 6:0:2:0: [sdc] Attached SCSI disk
[  +0.000697]  sdh: sdh1 sdh9
[  +0.000525] sd 6:0:7:0: [sdh] Attached SCSI disk
[  +0.004321]  sdg: sdg1 sdg9
[  +0.000289]  sde: sde1 sde9
[  +0.000028] sd 6:0:6:0: [sdg] Attached SCSI disk
[  +0.000329] sd 6:0:4:0: [sde] Attached SCSI disk
[  +0.002716]  sdf: sdf1 sdf9
[  +0.000341] sd 6:0:5:0: [sdf] Attached SCSI disk
[  +0.226387] ata3: SATA link down (SStatus 0 SControl 300)
[  +0.311218] ata4: SATA link down (SStatus 0 SControl 300)
[  +0.474542] ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[  +0.002091] ata5.00: ATA-9: Intenso SSD Sata III, W0406A0, max UDMA/133
[  +0.001001] ata5.00: 1000215216 sectors, multi 1: LBA48 NCQ (depth 32), AA
[  +0.004154] ata5.00: configured for UDMA/133
[  +0.000463] scsi 4:0:0:0: Direct-Access     ATA      Intenso SSD Sata 6A0  PQ: 0 ANSI: 5
[  +0.000921] sd 4:0:0:0: Attached scsi generic sg9 type 0
[  +0.000067] sd 4:0:0:0: [sdj] 1000215216 512-byte logical blocks: (512 GB/477 GiB)
[  +0.000644] sd 4:0:0:0: [sdj] Write Protect is off
[  +0.000216] sd 4:0:0:0: [sdj] Mode Sense: 00 3a 00 00
[  +0.000027] sd 4:0:0:0: [sdj] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[  +0.000455] sd 4:0:0:0: [sdj] Preferred minimum I/O size 512 bytes
[  +0.002323]  sdj: sdj1 sdj9
[  +0.000242] sd 4:0:0:0: [sdj] Attached SCSI disk
[  +0.305466] ata6: SATA link down (SStatus 0 SControl 330)
[Dec 1 21:07] EXT4-fs (dm-1): mounted filesystem d1bcb39a-1754-4d14-8bb7-59fe8a5ad4d8 ro with ordered data mode. Quota mode: none.
[  +0.000405] systemd[1]: custom-zfs-load-keys.service: Found dependency on systemd-journal-catalog-update.service/start
[  +0.000426] systemd[1]: custom-zfs-load-keys.service: Job systemd-journal-catalog-update.service/start deleted to break ordering cycle starting with custom-zfs-load-keys.service/start
[  +0.001671] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[  +0.046343] EXT4-fs (nvme0n1p2): mounted filesystem bf33ca0e-5d33-440d-a682-8d6349d83f13 r/w with ordered data mode. Quota mode: none.
[  +1.053048] audit: type=1400 audit(1764619742.033:3): apparmor="DENIED" operation="change_onexec" class="file" info="label not found" error=-2 profile="unconfined" name="snap-update-ns.lxd" pid=5764 comm="snap-confine"
[  +0.000044] sd 6:0:3:0: [sdd] tag#2190 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=2s
[  +0.000008] sd 6:0:3:0: [sdd] tag#2190 Sense Key : Medium Error [current] [descriptor]
[  +0.000006] sd 6:0:3:0: [sdd] tag#2190 Add. Sense: Unrecovered read error
[  +0.000006] sd 6:0:3:0: [sdd] tag#2190 CDB: Read(16) 88 00 00 00 00 04 0c 87 2b 38 00 00 01 00 00 00
[  +0.000004] critical medium error, dev sdd, sector 17390054432 op 0x0:(READ) flags 0x4000 phys_seg 1 prio class 0
[  +0.786016] zio pool=tank vdev=/dev/disk/by-id/scsi-SATA_TOSHIBA_MG09ACA1_X3M0A01CFJDH-part1 error=61 type=1 offset=8903705915392 size=1036288 flags=1074267304
[Dec 2 07:02] audit: type=1400 audit(1764655366.277:5): apparmor="DENIED" operation="change_onexec" class="file" info="label not found" error=-2 profile="unconfined" name="ubuntu_pro_apt_news" pid=2036386 comm="(python3)"
[  +0.005274] audit: type=1400 audit(1764655366.282:6): apparmor="DENIED" operation="change_onexec" class="file" info="label not found" error=-2 profile="unconfined" name="ubuntu_pro_esm_cache" pid=2036387 comm="(python3)"
[Dec 3 05:26] audit: type=1400 audit(1764736011.987:7): apparmor="DENIED" operation="change_onexec" class="file" info="label not found" error=-2 profile="unconfined" name="ubuntu_pro_apt_news" pid=2390548 comm="(python3)"
[  +0.007625] audit: type=1400 audit(1764736011.995:8): apparmor="DENIED" operation="change_onexec" class="file" info="label not found" error=-2 profile="unconfined" name="ubuntu_pro_esm_cache" pid=2390549 comm="(python3)"
[Dec 4 09:22] audit: type=1400 audit(1764836558.227:9): apparmor="DENIED" operation="change_onexec" class="file" info="label not found" error=-2 profile="unconfined" name="ubuntu_pro_apt_news" pid=2694719 comm="(python3)"
[  +0.007139] audit: type=1400 audit(1764836558.234:10): apparmor="DENIED" operation="change_onexec" class="file" info="label not found" error=-2 profile="unconfined" name="ubuntu_pro_esm_cache" pid=2694720 comm="(python3)"
[  +0.000003] audit: type=1400 audit(1764943932.512:27): apparmor="DENIED" operation="change_onexec" class="file" info="label not found" error=-2 profile="unconfined" name="ubuntu_pro_apt_news" pid=3189133 comm="(python3)"
[  +0.004146] audit: type=1400 audit(1764943932.516:28): apparmor="DENIED" operation="change_onexec" class="file" info="label not found" error=-2 profile="unconfined" name="ubuntu_pro_esm_cache" pid=3189135 comm="(python3)"
[Dec 6 10:02] audit: type=1400 audit(1765011758.963:29): apparmor="DENIED" operation="change_onexec" class="file" info="label not found" error=-2 profile="unconfined" name="ubuntu_pro_apt_news" pid=3368845 comm="(python3)"
[  +0.005244] audit: type=1400 audit(1765011758.968:30): apparmor="DENIED" operation="change_onexec" class="file" info="label not found" error=-2 profile="unconfined" name="ubuntu_pro_esm_cache" pid=3368846 comm="(python3)"

Function · December 6, 2025, 3:31pm

Do you know if the version of ZFS you’re using does sequential resilvering?

I read that link you sent but don’t know how I can tell, but since it says something about 2016 and I’m running zfs-2.2.2-0ubuntu9.2 which is from ~2023 (when I built this server) I guess I’m good on that end.

Does that indicate problems with two drives or do those both refer to the same drive?

Two drives. In my post above I explained how shortly after I replaced the first drive, a second drive failed, which is why any further failure until the resilver completes will make me lose this RAID-Z2 pool.

Edit: I want to make it clear that the speed has been going downhill mostly after the second drive failure. It’s not like the speed was good, then the second drive failed, and it’s been a constant slow speed since. Instead, it has always been going down, hour after hour. It’s 246M/s as of writing. I’m losing like 2/Ms per hour recently.

deviantintegral · December 6, 2025, 11:58pm

How full is your pool? I’ve only done basic trials with RAIDZ, but won’t a resilver be much slower over sections of the drives that are actually filled with data? That could explain the declining averages over time.

If there were additional trouble with hardware, you’d see messages like you see with sdd in your dmesg logs.

The feels like just the typical “why raidz is not always the best choice for large pools” scenario. I’ve always done a stripe of mirrors to avoid this (at the cost of less usable space). But given the drives are operating at ~1.5MB/sec and getting a few hundred IOPS, I think you may just have to wait for the rebuild to finish.

could this be a symptom of high fragmentation in the pool too? That would also explain high IOPS with low data rates.

Function · December 7, 2025, 11:55am

How full is your pool?

$ zpool list -o name,capacity,alloc,size tank
NAME    CAP  ALLOC   SIZE
tank    87%   114T   131T

The feels like just the typical “why raidz is not always the best choice for large pools” scenario.

I tried to educate myself before building the server and the sentiment in forums seemed to me like 8 drives was on the higher end recommended for RAID-Z but still manageable. I thought I would be fine if I have 2 out of 8 drives for parity. I only remember recommendations for the number of drives in the pool, not their capacity (in my case 18TB per drive).

I’ve always done a stripe of mirrors to avoid this

If I can afford it, I’d like to go with a mirror configuration with a bit more failure resilience in the future, since AFAIK a stripe of 2 mirrors can fail after just 2 failed drives (though not very likely). A stripe of 3 mirrors is very safe and fast but extremely costly, uses a lot of power, at only 1/3 of usable space. I might look into a 2x mirror of RAID-Z1 if that makes sense, but I’ll research if it makes sense before building it in the future.

But given the drives are operating at ~1.5MB/sec and getting a few hundred IOPS, I think you may just have to wait for the rebuild to finish.

I’ve ran some numbers and my current estimate with the slowdown is that it’s gonna take 60-90 days to finish if the trend continues.

some numbers (charts)

could this be a symptom of high fragmentation in the pool too?

$ zpool list -o fragmentation tank
 FRAG
  35%

I think 35% is moderate fragmentation.

Current status:

        97.9T / 114T scanned at 210M/s, 97.8T / 114T issued at 210M/s
        12.0T resilvered, 85.52% done, 22:59:53 to go