ZFS performance tuning for usenet

[professionaltart]

Wanted to throw it out there, given ZFS Performance Tuning for BitTorrent - OpenZFS - Practical ZFS exists - are there any difference in approach that should be taken when using usenet to download/build binaries instead of torrents, wrt things like record sizes, or if it’s still acceptable to download straight to a rust pool instead of an intermediate flash pool?

I have some test pools setup with spare disks (16TB SATA spinners, 100GB SAS SSDs) while deciding how I want to setup my pools and datasets for a final fileserver migration^, but all I’m currently testing with is robocopying a bunch of different folders/filetypes from a Windows host - not doing anything with torrents or nzbs themselves. My concerns are probably around how a torrent client can generally pre-allocate the file and then fill in the data as it obtains it, whereas this doesn’t seem to be something eg sabnzbd can do.

^My nominal thinking was to have things download to the high write-endurance SAS drives (probably just a big stripe set) while downloading, and then once complete move them onto the HDD pool (probably raidz2 to optimise for storage space, with eg either rs=1MB or whatever this translates to when striped across the vdev) for longer-term storage, with the hope of keeping files mostly contiguous without worrying about fragmentation/weird interleaving issues making a mess of the inherent poor iops later on. Especially given sabnzdb tends to sometimes leave a bit of a mess with downloads it starts but can’t find all the pieces to finish.

[mercenary_sysadmin]

@professionaltart

My concerns are probably around how a torrent client can generally pre-allocate the file and then fill in the data as it obtains it, whereas this doesn’t seem to be something eg sabnzbd can do.

There is no preallocation under ZFS. There can’t be, because it’s a copy on write filesystem. “Pre allocation” therefore means one of two things–depending on the technique used, ZFS either says “sure, why not” while doing nothing (in which case blocks and therefore sectors are allocated upon write, as normal) or ZFS actually is forced to allocate sectors immediately and treat them as any other data… In which case, new sectors are allocated later when the file is actually written to for real, because see above: copy on write filesystem; blocks (and therefore sectors) on disk are immutable and therefore never overwritten in place.

@professionaltart

Wanted to throw it out there, given ZFS Performance Tuning for BitTorrent - OpenZFS - Practical ZFS exists - are there any difference in approach that should be taken when using usenet to download/build binaries instead of torrents, wrt things like record sizes, or if it’s still acceptable to download straight to a rust pool instead of an intermediate flash pool?

Nah, same guidelines apply. Your Usenet binary encoded messages should typically be in the 768KiB to 1MiB size apiece, so you’ll want recordsize=1M. You should make certain to turn compression on, though–because even though the individual pieces are generally UUENCODEd versions of incompressible binaries, in UUENCODEd form they’re highly compressible. And since a lot of them will be 768K instead of 1M, you’ll want to make sure that slack space can be compressed as well.