When do you decide a drive is bad enough to replace?

OpenZFS
1

When do you personally decide that it’s time to replace a “bad” drive?

Previously i have replaced my drives as soon as i get any amount of data having to be fixed by zfs.
But it hit me today that maybe I’m being dumb and wasting my money?
Since i run 2 disk mirrors I can technically afford one drive to fail completely and even if anything would happen when i only have that one disk left i have a backup system (not that i necessarily want to spend the time restoring from that backup just because of being lazy).

So when do you guys replace your drives?
What are you monitoring, what are the signs that you are looking for?
Zfs errors? Smart? Complete drive failure?

2

I replace latest when I see ZFS errors, except when I determine that the ZFS errors are caused by a bad controller or cable rather than the drive.

Keep an eye on reallocated sector numbers in SMART. If they are increasing rapidly then the drive is very likely about to fail, even if the errors are not hitting the ZFS layer yet.

Keeping the time your RAID is degraded short is crucial for uptime, because while it’s degraded it’s statistically a(n) order(s) of magnitude more vulnerable to failing completely. If you don’t care about downtime and don’t mind to restore from backup then it doesn’t really matter.