Example ZFS on USB rig that survived any load I threw at it
ZFS on USB have worked great for me after I eliminated all apparent issues with the USB path. That’s included testing different enclosures based on different chipsets, hubs, and host controllers. The above is a pic of one of those iterations. Here’s some takeaways:
USB host side
- Intel host controllers are generally problem-free
- AMD on-CPU host controllers are not usable for this purpose
- AMD on-chipset host controllers are generally problem-free
- AMD ATX boards typically expose both and their manual states which are which
- AMD-based mini PCs might expose only on-CPU ports and they wouldn’t tell you
Cables
- Get decent cables and test them
USB device side (Oh boy)
- Ensure your enclosures are using known good chipsets
- This is generally true for WD external drives
- NST-370A31-BK is the same as S351BU313, both use ASM235CM and are problem-free under heavy sustained load
- Firmware can make a chipset behave well in one enclosure while being problematic in another. E.g. JMicron is generally unreliable in noname enclosures while perfectly stable in WD MyBook
- Ensure your enclosures keep them chipsets cool
- This is NOT true for WD external drives. WD uses both JMicron and ASMedia and I’ve had both overheat under enough sustained load
- You can fix existing enclosures by sticking small heatsinks to the chipsets. The ghetto server above has all 4 drives with heatsinked USB controllers, hole in the enclosure above the heatsink for air circulation. Here’s what that looks like on WD MyBook and WD Elements:
Shortcut
You can eliminate all variables but the host side by getting a USB box that someone verified sane. Here’s my contribution. The OWC Mercury Elite Pro Quad is a pretty sane design from what I can tell. It’s got good reputable USB-to-SATA chipsets, they’re cooled appropriately, they’re linked to a reputable USB hub chip and the thing comes with good cables. I have them hooked to 5Gbps USB ports and I’m observing 400-500MB/s from each box but they should be able to do double that on 10Gbps ports. I now have 4 in operation and not a single one has squeaked so far. You can read about the teardown and testing I’ve done in the thread I linked. They’re not cheap but they’re not stupidly expensive either - about $55 per bay, which in the ballpark of the decent single enclosures I mentioned.
A friend of mine has been inside a TERRAMASTER D6-320 too and I’ve seen the pics. It uses the same chips both for the bridges and the hub, although I think it’s got a second hub because it needs 6 ports. I haven’t tested it myself so I can’t vouch for it but if it’s the only thing you can get your hands on, it looks sane. It’s made in PRC while the OWC is Taiwanese if you care about that.