Using consumer SSDs for data with DC SSD for metadata?

Hi,

I was listening to the latest episode of 2.5 admins this morning, and I was wondering how to speed up vms on my home nas/server. One way would definitely be to use less spinning rust or atleast split off the data pool disks from the vm pool. Another would be to move to ssds, but I’m less than stoked on the price of high write endurance ssds.

But I was wondering - would it be possible to use eg. some consumer grade ssds for the data blocks, and use a mirror of DC grade ssds (with high write endurance) for the metadata? I’m not sure if this would be sufficient to offset the amount of writes the data ssds would get.

Looking at this a little differently, If this is a home nas, then why wouldn’t home quality ssd be appropriate? What kind of workload do you have that consumer ssd wouldn’t be fine? And they’re cheap enough you can have a spare on hand…

Cheap is relative, and mostly they’re not rated to handle the amount of writes ZFS can throw at a pool, I believe.

Don’t tell anyone … but in November of 2020 I bought a pair of Team Group GX2 512GB SSDs on a black Friday sale to replace some spinning rust I had been using to run some VMs. Rust was fine but boot performance when I bounced the host was poor, of course, and trying to install patches on all my VMs took ages while they fought for space in the queue. I bought THE cheapest SSDs I could find and paid $85 for the pair, shipped.

I had never heard of Team Group and had no expectation that they would last more than 3 minutes… But, I had a couple extra dollars at the time and wondered if it was even possible to get cheap disks like this to work for any length of time…

Fast forward to today and I am now running 11 Containers and 7 VMs on that same pair of SSDs. They are hosting Nextcloud, MariaDB, InfluxDB, Home Assistant, Nagios etc., etc.,

This is in my homelab, so I’m not winning any awards for raw speed but they vastly out perform the spinning rust and have never missed a beat. They are not enterprise class, they are not enterprise performance .. but they have worked fine for me.

Obviously, these disks were put into a ZFS mirror on day one, with compression enabled. I snap them every hour, and those snaps are sent to two offsite servers and a local replica server on the hour.

Like all of my storage, I don’t trust these disks. They get regular scrubs. And lots of backups, which I test.

The more confident I am in my backups the less I care about the individual disks in my arrays…

1 Like

Why would zfs be heavy on writes? It’s the same amount as of data.

Sure raid is a multiplier, but that’s true of any raid setup.

Am I missing something?

Metadata writing gets pretty intense, and sync writes generally get written out twice (once ephemerally to the ZIL, then permanently to main storage, then technically the deletion of the ephemeral copy in the ZIL is another write, although only a metadata write).

So yes, ZFS can actually produce more writes on the same workload as a conventional filesystem (although it typically gets through the workload faster, not slower, because those writes tend to be more efficient. Storage is complicated).

If this is a home nas, then why wouldn’t home quality ssd be appropriate?

They may be, or they may not be, depending on your workload and expectations. I’ve certainly used plenty of consumer SSDs with ZFS, and I frequently still do use consumer SSDs eg in desktops with ZFS. Even when that desktop runs a few VMs or what have you!

But that’s mostly because I frequently have a free or nearly free consumer SSD just lying around, even for the desktop use. If I have to buy something new, I go inexpensive enterprise pretty much every time, because they’re honestly a better deal IMO.

If I buy a Kingston DC600M, it’ll probably cost me about 20%-30% more than a similarly sized half-decent consumer SSD from a well known brand… But it will have double or in some cases four times larger write endurance, giving me full performance and reliability for as long as I care to use it, while the consumer SSD will be significantly slower within two to three years time, and utterly unbearable in six years.

Even without the performance benefits of, for example, hardware QoS on the enterprise gear to reduce latency, and even without any performance benefit when new due to higher throughput (which there frequently isn’t), and even without needing performance to sustain at high levels for long periods of heavy activity that would cripple the consumer gear… Even without any of that, I’d rather pay $390 for a drive that lasts me ten years at full performance, rather than $300 for a drive that’s at half speed before five years have elapsed.

Essentially, I think consumer SSDs are a sucker buy, in the same way that everything on the shelf at a Family Dollar is a sucker buy. The sticker price is certainly lower, yes, but it’s a very cynical worse value being pushed on you by a very cynical industry that understands the cuts it’s making much better than you do, and is perfectly willing to sell you the storage equivalent of a very pretty, moderately fast car with utterly terrible brakes if that’s how they can get a higher margin on more sales.

2 Likes

You could, but it wouldn’t be worth it. You might as well simply deal with the low write endurance; you’ll be replacing those consumer SSDs in the same number of years whether they’re receiving the metadata writes or not.

If you think that paying a perhaps 30% premium for double the write endurance is a bad deal, you can just use the consumer SSDs; as long as they’re decent (trash SSDs ARE a thing; no name crap you’ve never heard of before may be okay or it may be completely unfit for purpose, with or without ZFS) they’ll work fine with ZFS just like with any other filesystem.

But when you replace those drives with more “cheap” consumer SSDs down the road, you’ll have paid roughly half again as much as somebody with the same workload who bought better gear with double the write endurance and is still years away from their next drive replacement.

Essentially, consumer SSDs might make sense if you expect to outgrow your drives before they wear out. But if you plan to get the full lifespan out of your drives, they’re a horrible deal designed for suckers. :cowboy_hat_face::beer_mug:

It was just “an idea” I had. I’d rather pay to be certain that I can trust the medium (but, as always, have backups).

1 Like

I used to keep two pools: one 512G SSD pool, and one 2T rust pool. My OS and VMs went on the fast SSD pool, while bulk file storage–yanno, “Linux ISOs” and whatnot–went on the bigger and cheaper but much slower rust.

But then SSDs dropped in cost enough that I replaced both pools with a single 4T SSD mirror. Simpler, easier, faster.

The only place I use rust anymore is for huge backup servers. If the box in question has/needs less than about 50T of capacity, I just stick with SSDs. But the server that backs up every system in my home, plus a bunch of remote stuff? It’s got a 96T rust based pool.

I “only” have about 40TB raw storage, but replacing it with ssds is expensive to say the least :frowning: definitely when it’s enterprise grade storage.

Thank you for recommending the Kingston DC600M!

Reading the Kingston site, it appears that DC600M firmware updates are only supported through Windows. There is a mention of contacting support for other options. Do you know if DC600M firmware updates can be performed using Linux?

Not off the top of my head, no.

About a year ago, I bought a couple of 2TB m.2 nvme Samsung P983 SSDs off eBay. I was keeping costs down and they were about $100 each. They’ve been totally reliable until I removed my GPU. Seems like too much of a coincidence that it’s the GPU removal’s fault, but one of the SSDs keeps vanishing (zpool status = REMOVED). Reappears after a restart. I’m still troubleshooting but this thread caught my eye as I consider replacing the drive(s).

Would replacing an m.2 Samsung P983 with a DC600M impact performance much? SATA would theoretically be 25% as fast as my current drives but does that matter? I guess in nutshell, @mercenary_sysadmin or whoever has experience with them, are the DC600Ms fast enough for a bunch of VMs (6) and LXC (4) containers? They’d be connected to a LSI 9300-8i HBA along with 6 HDDs.

New DC600Ms would be more expensive but ten years of use would be sweet!

I really appreciate this site! Thank you everybody here!

I’ve got pools of six DC600M (in mirrors) backing KVM-served VMs running Windows server AD, Windows server filesharing, and MSSQL backed ERP apps servicing 500+ concurrent user teams, and kicking ass at it.

Note that if you mix DC600M and PM983 in the same pool, you’ll essentially get the lowest common denominator performance wise. But since the PM983s also offer PLP and QoS, that’s at least hopefully not a real issue. The biggest concern I have is that PM983s are notorious for overheating… And if your remaining PM983s thermally throttle badly, they’ll drag your pool performance right down with them.