ZFS & read/write performance in a world of cheap, very fast NVMe SSDs

[Cowlo]

I saw that the recent 2.3.0 release adds Direct IO support after a couple years of pre-merge back-and-forth:

OpenZFS 2.3.0 Release--Direct I/O Question; Also I don't understand how distros package ZFS, apparently?

See: Release zfs-2.3.0 · openzfs/zfs · GitHub One of the features is Direct I/O, which bypasses the ARC cache. Apparently, this gives big performance improvements for NVME pools? Is there a recommended article for explaining exactly how this works? I think I must not understand something, as I assumed RAM was always faster than NVME storage. Maybe that’s not always true anymore? Alternatively, does the overhead of moving data in and out of the ARC cache really cancel out RAM’s speed advantage…

I think there hasn’t yet been a lot of commentary about direct IO in this forum. Is that because direct IO is a fairly situational optimization?

This talk from a few years ago suggests the feature came about to try to improve the ZFS-on-NVMe story:

DirectIO for ZFS by Brian Atkinson