Impending Drive Failure?

OpenZFS
1

This is not a ZFS specific post, but wanted to seek out the opinions of hive minds as to whether or not I should replace the drive before it actual fails and borks my pool.

…
[142649.952676] sd 6:0:12:0: [sdde] tag#5269 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=2s
[142649.952696] sd 6:0:12:0: [sdde] tag#5269 Sense Key : Medium Error [current] [descriptor]
[142649.952706] sd 6:0:12:0: [sdde] tag#5269 Add. Sense: Unrecovered read error
[142649.952716] sd 6:0:12:0: [sdde] tag#5269 CDB: Read(16) 88 00 00 00 00 06 5d df fe 20 00 00 00 e0 00 00
[142649.952722] blk_update_request: critical medium error, dev sdde, sector 27344764624 op 0x0:(READ) flags 0x4200 phys_seg 4 prio class 0
[142649.975364] blk_update_request: critical medium error, dev dm-64, sector 27344764448 op 0x0:(READ) flags 0x0 phys_seg 15 prio class 0
…

Smartmontools does NOT report anything in the grown defects list or any re-allocated sectors but does show a segment error. I have never seen a segment error before but I presume this is not a simple matter of a few un re-allocated sectors but something more significant.

Given this drive actually shutdown one of my drive bays, twice, I think its a serious, non trivial matter hardware fault?

Num  Test              Status                 segment  LifeTime  LBA_first_err [SK ASC ASQ]

     Description                              number   (hours)

# 1  Background long   Failed in segment -->       7   56527          17987336 [0x3 0x5d 0x1]

Thanks in advance.

2

If you have the option, replace the drive. First priority should be the safety of the pool.

Once that risk is taken care of, you should run a long smart test on the drive (on a separate system) and also possibly just to a badblock test. Typically transient errors should disappear (or be fixed by the firmware) and in all probability, you should see that the drive itself is actually good.

From what you’ve said, the SMART on the drive seems good - check for Reallocated sectors, Pending sectors, Offline uncorrectable sectors and Reported uncorrectable sectors (hopefully all of these are reported by smartctl).

Based on the above, I think there was some sort of a hardware (not drive internal) issue that caused the trip up.

3

Thanks for the tip. I was just about to do that when the drive dropped suddenly dropped off-line kicked off an auto-rebuild. I will need to do further testing of the driver once the resilver is complete. The JBOB shut the port down after repeated read errors, which presumably is triggered by an error threshold inside of SES code. Same exact thing happened to the same drive in different enclosure so I know that its not slot related, the drive is presumably foo-barred and needs to go.