Clustering with ZFS

Another thread here got me thinking about ZFS and clustering options and I recall @mercenary_sysadmin mentioning in a 2.5 Admins episode that one could use DRBD and ZFS together to share storage among multiple nodes.

I’m curious about the particulars of this setup and would love to hear some details about how that might work.

1 Like

DRBD is just a virtual block device (Distributed Redundant Block Device) that can be fed with any arbitrary real block devices on the cluster nodes. There’s no particular magic involved here, you just feed DRBD the devices exposed by ZFS zvols rather than feeding them individual drives or mdraid arrays.

Now, there probably is a wealth of potential tips for performance tuning–but I can’t give you any, because I’ve never used DRBD in anger myself. I’d be very interested in hearing any war stories, though!

DRBD creates a block device that acts like a shared disk that multiple nodes can access at the same time. Since ZFS isn’t a “clustered” filesystem and cannot be mounted on multiple hosts simultaneously, if you put ZFS on top of DRBD, you would be limited to an active-passive configuration (per pool), like for backing an NFS server with a standby. If you put ZFS (zvols) below DRBD, then you need another filesystem to put on the created block device, possibly one that supports shared-disk clustering if you’re aiming to mount the filesystem on the multiple nodes simultaneously.

Thanks, both of you, for the responses; it’s really helpful to have that picture of how the layers can be stacked. I’m finding myself curious about these clustered file systems lately. Do you two (or does anyone) have any real-world favorites in this space? I’ve glanced at DRBD and Ceph. Anyone have any stories about using these or others that they’d care to share?

DRBD is pretty easy to set up, if your only real concern is “verified working,” and all you need is two systems.

Ceph is a multi-headed nightmare suitable for large numbers of individual systems. It distributes a lot further than DRBD does, but you’d better bring your A game, especially if you have individual-task performance concerns. And you’ll need more hardware before it’s worth it.

I set up a toy DRBD system once to play with it, but never went any further than that. I’ve seen Ceph in action, but have never set it up or maintained it myself.

1 Like