Redesigning pool with too wide vdevs

Hi all,

I have a question on re-designing zfs pool layout considering hardware prices..

In my work, previous admin have built a truenas machine with 2 seperate zpools, with 16 18TiB device each. Neither is even 1/3rd full.

As there is no short term plan to add much more data, I want to redesign the pools. Is my plan a
good one:

  • Replicate data from pool1 to pool2 (move the shares over and all those shenanigans)
  • Re-use disks from pool2, create a new pool with, 2x8-wide RAIDZ2 vdevs
  • Replicate back the data from pool2 to pool1.
  • Use 8 of pool1 disks to add one more vdev
  • Add 2 as spares
  • Keep the rest for future expansion and offline spare.

Like this I’ll have 3 v-devs with 8 disks, plus spares, and future expansion capability.

First question, sorry but I gotta ask it… Do you have backup of these pools?

yes, I have one daily zfs replication job from another TrueNas box (pull).

Also before doing all these, I can (and will) do a 3rd backup to another ZFS host.

Okay, great. With that out of the way, I’d recommend either six wide or ten wide Z2 instead of eight wide, depending on whether you value maximum storage efficiency for large files (and no database workloads!) or a mix of storage efficiency and better performance.

RAIDz spreads each block among all data members of each stripe. So, say you’re saving a 1MiB block to an eight wide Z2, you will end up with 1MiB/6 written to each disk. That doesn’t divide evenly, so you end up needing padding, which both reduces performance and storage efficiency from what you’d naively expect.

This isn’t the end of the world, and if you’re building a bulk file server with no DB workload on a simple 1GiB LAN, the performance issues will almost certainly be hidden behind the network bottleneck. BUT while it isn’t the end of the world, it DOES make a difference, so since you’re re-architecting in the first place, I’d suggest taking this into account.

Also note that with six or ten wide Z2, you really should be using a large recordsize. 1M or larger for six wide, 4M for ten wide. The other part of splitting each block between all data members of a stripe (unlike conventional raid, which writes to each disk in fixed chunk sizes, 512KiB by default on eg mdraid) is that if those individual writes per drive are too small, each drive individually will perform poorly.

With the default 128KiB recordsize and a ten wide Z2, you’d only be writing 128/8==16KiB per drive, which means most of your I/O will be 16K random–a punishing workload for rust and SSDs alike. With recordsize=4M, you’d instead be writing 4,096KiB/8==512KiB per drive, with massively better performance.

Remember when I said “assuming no database workloads” up above? If you run MySQL InnoDB on that same ten wide Z2, it doesn’t matter what recordsize you set, because InnoDB writes in 16KiB pages–each of which will be stored 4KiB per drive on only four data and two parity drives per stripe. This leaves you with 4KiB random I/O on each drive (the most punishing workload possible), much more space per write burned on parity than you would naively expect (67% storage efficiency instead of 80%), AND nearly half of your drives idle during any given operation, because the four drives not involved in the current write aren’t enough to issue a second read or write in parallel, even if you have something in the queue.

If you need to do much in the way of database or VM workloads, I STRONGLY recommend narrower vdevs for performance reasons–4-wide Z2, 3-wide Z1, or mirrors.

You’re also going to have enough drives in this pool that you might want to consider having a spare or two, to cut down on the vulnerability window after a drive failure occurs. I would ESPECIALLY recommend allocating a spare or two if you’re planning to do 3-wide Z1 or 2-wide mirrors.

Thanks for your quick answer Jim!

RAIDz spreads each block among all data members of each stripe. So, say you’re saving a 1MiB block to an eight wide Z2, you will end up with 1MiB/6 written to each disk. That doesn’t divide evenly, so you end up needing padding, which both reduces performance and storage efficiency from what you’d naively expect.

This is the kind of wisdom I’m here for… That makes a lot of sense.

It’s mostly long term storage or file sharing via Samba, NFS and webdav, no databases (at least yet). I like the idea of 6-wide Vdevs, RAIDZ1 makes me nervous in general :slight_smile:

This is good info, gets me thinking about my own redesign … I’m not quite in the OP scale of things though.

Current setup is 2 pools of 5x raidz1 12TB disks each. Will be moving that to a single pool of 3x vdevs of 4x raidz1 12TB disks, with a special vdev of 2x mirrored 512G nvme drives. The bulk of the storage is media, so their datasets are recordsize=1M and special_small_blocks=32K.

I ran pilot tests over recordsize/special_small_blocks, testing how much actually hit the special vdev and what the cold read times were - those were the best values.

Also checked Plex and Emby, and best for those was recordsize=128K and special_small_blocks=32K. My gut feel as 16K/16K, but that really didn’t perform any better, and cost more space. I haven’t yet checked if putting their sqlite3 DBs on dedicated datasets helps at all.

raidz1 is fine (for me) - I literally have the full content backed up onto multiple backup systems. Every file is on at least 3x and in most cases 4x separate systems, with snapshots going back a year. The main chassis is a 36-bay Supermicro CSE-847, so it’s easy to shuffle drives into it for local replication - much faster than via 10G network :wink:

The value of moving the sqlite to a separate dataset isn’t so much optimizing for the DBs, it’s freeing yourself up to optimize the media datasets themselves, with far larger recordsize than you’d be able to get away with if the sqlite has to share the same settings.

With that said, I seriously doubt you actually need to fully minmax the storage tuning for a personal Plex service. :cowboy_hat_face:

I’d strongly suggest 4x 3-wide Z1 instead. See above discussion re: when you need to add padding to every block or not. You’ll effectively get about the same actual storage efficiency out of the pool (because losing the padding will gain you back most of not all of the theoretical SE you lost by going narrower), and performance will be considerably better–especially since those viewable Linux ISOs are already compressed.

You also pick up significantly better fault tolerance, and although you’re unlikely to much care about performance in normal day to day use, resilvers will complete significantly faster. It’s a win all the way around.

The Plex and Emby datasets are just for those services - they’re separate from the media itself. And yup, doesn’t need to be super performant - Plex is just there for my daughter to get remote access to the media :slight_smile:

I had a sneaking suspicion you (or someone) would suggest 4x vdevs of 3x disks … Given that it’s mostly media storage, I went for capacity. The zfs calculator (https://www.45drives.com/zfs/zfs-calculator/) shows 3x vdevs of 4x disks is 130T raw, 95T usable, while 4x vdevs of 3x disks is 130T raw but 85T usable.

You’re saying they’re about equal though ?

Yes I am. Those calculators are all naive at best, and frequently downright batshit crazy. The padding is going to eat pretty much the entire difference between the naive 75% and 67% storage efficiencies you’re expecting.

When you store a 128KiB block on three data drives and one parity drive per stripe, you burn about 10KiB on padding and 44KiB on parity. This gives you a less naive expectation of 128KiB/(128KiB+10KiB+44KiB)~=68.8%.

When you store the same block on two data drives and one parity drive per stripe, you need no padding and burn 64KiB on parity. This gives you a less naive expectation of 128KiB/(128KiB+64KiB)~=66.7%.

These are still less naive estimations, not perfect ones, because we’re not talking about metadata or other small blocks… Which are also more efficiently stored on the 3-wide Z1. But you get the idea.

But we’ve already arrived at a MUCH closer situation than naive calculators are going to show you. Literally all those things are doing is dividing the data members per stripe by the total members per stripe and calling it a day.

I’ve got about 48T of media - movies, TV and 8 years of Division 1 volleyball games (daughter had a hellacious school/college/travel-team volleyball career- sheesh). All that stuff is multi-gigabyte files, currently transferring from the 2x pools of 5x raidz1 disks to the current new 3x vdevs of 4x disks using 1M recordsize and 32k special_small_blocks. Then there’s a TON of other datasets of random stuff, all 128k recordsize. And the Plex/Emby datasets (36G ea) with 128k/32k.

Before I kill the transfers and rebuild the new pool as 4x vdevs of 3x disks … we can expect the same treatment with the bulk data ?

Yeah pretty much. But you can find out for sure with toy pools.

Truncate 4x 8GiB files. Make a toy pool with a single 4-wide Z1 out of them. Set ashift and recordsize to the same values you’ll use in production. Copy a single 4GiB incompressible file to it (eg movies). Check and save output of zfs list on that toy pool.

Now destroy the pool, create a 3-wide Z1 using three of the same files (or destroy the old four and truncate yourself three fresh ones, whichever makes you happier). Set ashift and recordsize to the planned production values again. Copy the exact same 4GiB file to it, and once again check and save output of zfs list against the toy pool.

Now destroy the second toy pool, delete the truncated files, and compare the saved outputs of thezfs list commands you ran earlier. How much used and how much free usable space did each pool report remaining?

You can take percentages on this, then scale up from a single vdev of 8GiB “drives” to three or four vdevs of 18T drives. This still isn’t a down to the last block accurate prediction, but it’s at least storing real data using the same actual algorithms and the same implementation of those algorithms as you’ll be using on real metal in the scaled up version. MUCH closer to reality than those silly calculators are, IME.

With a 21G movie file …

$ mkdir zfs-test
$ cd zfs-test
$ for f in 1 2 3 4 ; do truncate -s 12G disk${f}.img ; done
$ sudo zpool create -o ashift=12 -o autotrim=on -o compatibility=openzfs-2.1-linux -O acltype=posixacl -O compression=lz4 -O atime=off -O xattr=sa -O dnodesize=auto wide4 raidz1 /home/alice/zfs-test/disk{1,2,3,4}.img
$ sudo zfs create -o recordsize=1M -o mountpoint=/home/alice/zfs-test/wide4 wide4/wide4
$ sudo chown alice:alice wide4

$ zpool list -v wide4
NAME                                 SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
wide4                               47.5G   840K  47.5G        -         -     0%     0%  1.00x    ONLINE  -
  raidz1-0                          47.5G   840K  47.5G        -         -     0%  0.00%      -    ONLINE
    /home/alice/zfs-test/disk1.img    12G      -      -        -         -      -      -      -    ONLINE
    /home/alice/zfs-test/disk2.img    12G      -      -        -         -      -      -      -    ONLINE
    /home/alice/zfs-test/disk3.img    12G      -      -        -         -      -      -      -    ONLINE
    /home/alice/zfs-test/disk4.img    12G      -      -        -         -      -      -      -    ONLINE

$ zfs list -t all -r wide4
NAME          USED  AVAIL     REFER  MOUNTPOINT
wide4         994K  33.4G      140K  /wide4
wide4/wide4   140K  33.4G      140K  /home/alice/zfs-test/wide4

$ cp -av /src/Ghosted wide4

$ zfs list -t all -r wide4
NAME          USED  AVAIL     REFER  MOUNTPOINT
wide4        19.8G  13.6G      140K  /wide4
wide4/wide4  19.8G  13.6G     19.8G  /home/alice/zfs-test/wide4

$ sudo zpool destroy wide4

-----------------------------------------------------------------

$ sudo zpool create -o ashift=12 -o autotrim=on -o compatibility=openzfs-2.1-linux -O acltype=posixacl -O compression=lz4 -O atime=off -O xattr=sa -O dnodesize=auto wide3 raidz1 /home/alice/zfs-test/disk{1,2,3}.img
$ sudo zfs create -o recordsize=1M -o mountpoint=/home/alice/zfs-test/wide3 wide3/wide3
$ sudo chown alice:alice wide3

$ zpool list -v wide3
NAME                                 SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
wide3                               35.5G  1.36M  35.5G        -         -     0%     0%  1.00x    ONLINE  -
  raidz1-0                          35.5G  1.36M  35.5G        -         -     0%  0.00%      -    ONLINE
    /home/alice/zfs-test/disk1.img    12G      -      -        -         -      -      -      -    ONLINE
    /home/alice/zfs-test/disk2.img    12G      -      -        -         -      -      -      -    ONLINE
    /home/alice/zfs-test/disk3.img    12G      -      -        -         -      -      -      -    ONLINE

$ zfs list -t all -r wide3
NAME          USED  AVAIL     REFER  MOUNTPOINT
wide3         927K  22.9G      128K  /wide3
wide3/wide3   128K  22.9G      128K  /home/alice/zfs-test/wide3

$ cp -av /src/Ghosted wide3

$ zfs list -t all -r wide3
NAME          USED  AVAIL     REFER  MOUNTPOINT
wide3        20.4G  2.51G      128K  /wide3
wide3/wide3  20.4G  2.51G     20.4G  /home/alice/zfs-test/wide3

I did it again using the same 3x vdevs of 4x disks, and 4x vdevs of 3x disks

mkdir zfs-test                                                                                                                                                                                                
cd zfs-test                                                                                                                                                                                                   
for f in $(seq 01 12) ; do truncate -s 4G disk${f} ; done

sudo zpool create wide4 -o ashift=12 -o autotrim=on -o compatibility=openzfs-2.1-linux -O acltype=posixacl -O compression=lz4 -O atime=off -O xattr=sa -O dnodesize=auto raidz1 /home/alice/zfs-test/disk{1,2,3,4} raidz1 /home/alice/zfs-test/disk{5,6,7,8} raidz1 /home/alice/zfs-test/disk{9,10,11,12}
sudo zfs create -o recordsize=1M -o mountpoint=/home/alice/zfs-test/wide4 wide4/wide4
sudo chown alice:alice wide4

zpool list -v wide4
NAME                              SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
wide4                            46.5G  1.43M  46.5G        -         -     0%     0%  1.00x    ONLINE  -
  raidz1-0                       15.5G   520K  15.5G        -         -     0%  0.00%      -    ONLINE
    /home/alice/zfs-test/disk1      4G      -      -        -         -      -      -      -    ONLINE
    /home/alice/zfs-test/disk2      4G      -      -        -         -      -      -      -    ONLINE
    /home/alice/zfs-test/disk3      4G      -      -        -         -      -      -      -    ONLINE
    /home/alice/zfs-test/disk4      4G      -      -        -         -      -      -      -    ONLINE
  raidz1-1                       15.5G   472K  15.5G        -         -     0%  0.00%      -    ONLINE
    /home/alice/zfs-test/disk5      4G      -      -        -         -      -      -      -    ONLINE
    /home/alice/zfs-test/disk6      4G      -      -        -         -      -      -      -    ONLINE
    /home/alice/zfs-test/disk7      4G      -      -        -         -      -      -      -    ONLINE
    /home/alice/zfs-test/disk8      4G      -      -        -         -      -      -      -    ONLINE
  raidz1-2                       15.5G   472K  15.5G        -         -     0%  0.00%      -    ONLINE
    /home/alice/zfs-test/disk9      4G      -      -        -         -      -      -      -    ONLINE
    /home/alice/zfs-test/disk10     4G      -      -        -         -      -      -      -    ONLINE
    /home/alice/zfs-test/disk11     4G      -      -        -         -      -      -      -    ONLINE
    /home/alice/zfs-test/disk12     4G      -      -        -         -      -      -      -    ONLINE

zfs list -t all -r wide4
NAME          USED  AVAIL     REFER  MOUNTPOINT
wide4         942K  32.7G      140K  /wide4
wide4/wide4   140K  32.7G      140K  /home/alice/zfs-test/wide4

cp -av /src/Ghosted wide4

zfs list -t all -r wide4
NAME          USED  AVAIL     REFER  MOUNTPOINT
wide4        19.8G  12.9G      140K  /wide4
wide4/wide4  19.8G  12.9G     19.8G  /home/alice/zfs-test/wide4

sudo zpool destroy wide4

------------------------------------------

sudo zpool create wide3 -o ashift=12 -o autotrim=on -o compatibility=openzfs-2.1-linux -O acltype=posixacl -O compression=lz4 -O atime=off -O xattr=sa -O dnodesize=auto raidz1 /home/alice/zfs-test/disk{1,2,3} raidz1 /home/alice/zfs-test/disk{4,5,6} raidz1 /home/alice/zfs-test/disk{7,8,9} raidz1 /home/alice/zfs-test/disk{10,11,12}
sudo zfs create -o recordsize=1M -o mountpoint=/home/alice/zfs-test/wide3 wide3/wide3
sudo chown alice:alice wide3

zpool list -v wide3
NAME                              SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
wide3                              46G  1.43M  46.0G        -         -     0%     0%  1.00x    ONLINE  -
  raidz1-0                       11.5G   456K  11.5G        -         -     0%  0.00%      -    ONLINE
    /home/alice/zfs-test/disk1      4G      -      -        -         -      -      -      -    ONLINE
    /home/alice/zfs-test/disk2      4G      -      -        -         -      -      -      -    ONLINE
    /home/alice/zfs-test/disk3      4G      -      -        -         -      -      -      -    ONLINE
  raidz1-1                       11.5G   344K  11.5G        -         -     0%  0.00%      -    ONLINE
    /home/alice/zfs-test/disk4      4G      -      -        -         -      -      -      -    ONLINE
    /home/alice/zfs-test/disk5      4G      -      -        -         -      -      -      -    ONLINE
    /home/alice/zfs-test/disk6      4G      -      -        -         -      -      -      -    ONLINE
  raidz1-2                       11.5G   280K  11.5G        -         -     0%  0.00%      -    ONLINE
    /home/alice/zfs-test/disk7      4G      -      -        -         -      -      -      -    ONLINE
    /home/alice/zfs-test/disk8      4G      -      -        -         -      -      -      -    ONLINE
    /home/alice/zfs-test/disk9      4G      -      -        -         -      -      -      -    ONLINE
  raidz1-3                       11.5G   384K  11.5G        -         -     0%  0.00%      -    ONLINE
    /home/alice/zfs-test/disk10     4G      -      -        -         -      -      -      -    ONLINE
    /home/alice/zfs-test/disk11     4G      -      -        -         -      -      -      -    ONLINE
    /home/alice/zfs-test/disk12     4G      -      -        -         -      -      -      -    ONLINE

zfs list -t all -r wide3
NAME          USED  AVAIL     REFER  MOUNTPOINT
wide3         831K  29.7G      128K  /wide3
wide3/wide3   128K  29.7G      128K  /home/alice/zfs-test/wide3

cp -av /src/Ghosted wide3

zfs list -t all -r wide3
NAME          USED  AVAIL     REFER  MOUNTPOINT
wide3        20.0G  9.64G      128K  /wide3
wide3/wide3  20.0G  9.64G     20.0G  /home/alice/zfs-test/wide3

sudo zpool destroy wide3

The part to pay attention to there is the difference in raw capacity. You burned an extra 12GiB raw on the 4w in comparison to the 3w.

So, after storing 21GiB of data on the 4-wide Z1, we have 19.8GiB used plus an estimated 13.6GiB remaining. Add those up and divide by the raw 48GiB capacity, and we have an experienced storage efficiency of roughly 33.4GiB/48GiB == 69.6%.

Now we do the same for the 3-wide Z1, and (20.4GiB + 2.5GiB)/36GiB == 62.8%.

Now, there is still a problem here. I said “estimated” free space, because that’s all ZFS can do. Compression, metadata, small files, and more can all change how much data you can really fit into a given amount of raw storage.

But this is still better than relying on a calculator, and–despite my hedging about free space estimation–you’ll notice that our experienced delta of 6.8% is significantly less than the 8.3% that a naive comparison of 3/4 and 2/3 would lead you to expect.

Providing yet another perspective to look at that number, the delta between storage efficiencies is already 18.1% smaller than the delta we expected… Despite having an entirely ideal storage workload consisting of between one and a tiny handful of files, eating 21GiB with essentially nothing in the way of metadata or other small blocks to handicap the wider vdev further.

I suspect that number would shrink even further, if you forced the 4-wide Z1 to store enough more data that each vdev was about the same ratio of USED/AVAIL.

But anyway, you can see where that’s headed. With vdevs already this narrow, sure, you do still save some space with the wider vdev, despite the padding. The question is, is 6.8% of your raw capacity really worth a layout with significantly slower resilvering, less fault tolerance, and more larger drives needed at once to expand capacity in the future really worth it?

144TB raw == 131TiB. 131TiB * .069 == 9.0TiB.

Personally, I don’t think that extra 9TiB is worth what you have to pay to get it. But if you disagree with me, at least now you have a better understanding of why you do, and by how much… and also just how much less concrete the numbers you’re playing with actually are than they initially appear. :clinking_beer_mugs::cowboy_hat_face::+1: