ZFS over iSCSI Storage Type Device in Proxmox is not Compatible with TrueNAS's iSCSI Target Protocols … But That Might Change Soon: A Summary of Events

Proxmox
1

@adaptive_chance asked for more info on the state of using ZFS over iSCSI in Proxmox with a target on TrueNAS SCALE/Community Edition. Here’s a brief summary based on what I understand so far. :slight_smile:

This is specifically for ZFS over iSCSI in Proxmox, which allows you to mount an iSCSI target from Proxmox and use it as a ZFS storage device–allowing, e.g., storing RAW virtual disk images that can be properly snapshotted, among other benefits c.f. NFS-backed storage. (This is separate from per-VM individual iSCSI block devices.) Here’s the Feature Request: Add LIO or IET iSCSI target compatibility to SCALE so Proxmox "ZFS over ISCSI" can work natively - Feature Requests - TrueNAS Community Forums

Problem/Justification
SCALE doesn’t implement LIO or IET which are required by Proxmox. Storage: ZFS over ISCSI - Proxmox VE

There is a workaround by using a sometimes buggy API. The ask here is to allow Proxmox to more natively be integrated into TrueNAS.

LIO or IET are iSCSI target protocols that TrueNAS doesn’t support.

From that thread, an iX Systems employee:

image
image1660×456 43.6 KB

I don’t understand why, but apparently implementing either one of these would break too much internally to be practicable.

Meanwhile, on the Proxmox side, the community plugin that’s maintained by PVE users to indirectly implement ZFS over iSCSI via API calls and SSH root access is currently being rewritten to squash some bugs and get around the old TrueNAS API being depreciated (and breaking) and the new API websockets being buggy in TrueNAS 25.04.0. Something about depreciating the API also broke the plugin in 24.10.2, so that also had to be fixed.

iX has indicated that the old method that the FOSS project uses (just logging in via SSH and executing raw ZFS commands) is a non-starter going forward and needs to be replaced with the correct API calls, but the latest word on the GitHub for the FOSS project is that the API doesn’t implement everythng they need to actually do that, so they’re on to … plan C at this point.

It keeps getting more complicated, but they’re not giving up. It’s spawned a couple of JIRA issues to get the TrueNAS API and documentation for the API fixed up a bit.

I don’t know enough about iSCSI target technologies to have an opinion about iX not wanting to implement the target protocols that Proxmox actually supports, but they’re not into doing that, so we’re taking the long way around the mountain.

Here’s the Proxmox issue: Making sure you're not a bot!
And the 3rd party FOSS plugin currently being worked on to get it working again with the 25.04 API: Truenas Scale 2025.04 api error · Issue #205 · TheGrandWazoo/freenas-proxmox · GitHub

EDIT:
Apparently, there’s been a breakthrough on the 25.04 websocket-based API compatible plugin.

image
image1866×316 30.3 KB

The community over on the Freenas-Proxmox GitHub has really been tearing into this over the last couple weeks. It’s been awesome to see. At this rate, iX might not have to do anything to get this working correctly.

2

I have a sneaking curmudgeonly suspicion that iX doesn’t really want TrueNAS Scale to interoperate with Proxmox, because iX is getting quite a lot deeper into Proxmox’s own territory with the move from Core to Scale… not to mention the recently-announced deprecation of Core in its entirety.

3

I think it’s out of iX’s hands at this point.

The community wants it, and is building their own plugin for Proxmox to do it.

Again. After the last plugin got broken by a TrueNAS update.

4

Well, here’s hoping for the best outcome for the end users, and that’s all I’ll say about that in either direction. :slight_smile:

2 Likes
5

Just read both of those threads. The community is trying to get a management integration working so that PVE can control/manage the back-end storage. iSCSI itself is oblivious to all this.

If PVE can do iSCSI against a “real” storage array (e.g. EMC, Pure, NetApp) then it can use a TrueNAS system as well. iSCSI is iSCSI is iSCSI – it’s either compliant with RFC7143 or it’s not iSCSI.

Maybe if I get some time this week I’ll spin up a PVE VM and a TrueNAS Scale VM and see what the ruckus is about.

6

If PVE can do iSCSI against a “real” storage array (e.g. EMC, Pure, NetApp) then it can use a TrueNAS system as well. iSCSI is iSCSI is iSCSI – it’s either compliant with RFC7143 or it’s not iSCSI.

The issue here is, unfortunately, not a lack of compliance with the iSCSI standard itself, exactly. It took a while to unravel exactly why this is so complicated. A few points:

  1. There are multiple standards/protocols to talk to an iSCSI target. Proxmox’s builtin “ZFS over iSCSI” storage type implements a subset of them. See: Storage: ZFS over ISCSI - Proxmox VE


This backend accesses a remote machine having a ZFS pool as storage and an iSCSI target implementation via ssh. For each guest disk it creates a ZVOL and, exports it as iSCSI LUN. This LUN is used by Proxmox VE for the guest disk.

The following iSCSI target implementations are supported:

    LIO (Linux)

    IET (Linux)

    ISTGT (FreeBSD)

    Comstar (Solaris)
  1. First issue: TrueNAS uses an iSCSI target implementation that is not on that list, and has no plans to implement one of the target implementations on the list.

Result: An unofficial, separate TrueNAS-compatible iSCSI storage plugin for Proxmox was created and maintained by TheGrandWazoo on GitHub.

  1. Second issue: iX does not officially support third party software accessing TrueNAS and running ZFS commands. It bypasses the TrueNAS “middleware,” which means TrueNAS itself isn’t aware of changes being made to the underlying ZFS pool, which can lead to unpredictable results. As much as possible, the TrueNAS API must be used to access the middleware to do ZFS operations.
  2. Third issue: TrueNAS 24.10/25.04 depreciated the old API that the previous Proxmox TrueNAS plugin used, breaking TheGrandWazoo’s plugin.

Result: Using TheGrandWazoo’s plugin as inspiration, a new project has implemented a TrueNAS-compatible ZFS-over-iSCSI plugin for Proxmox that is compatible with TrueNAS 24.10 and 25.04 and uses the current TrueNAS API. It’s currently in alpha testing, and not quite feature complete–a bit more work needs to be done to finish stripping out the SSH commands.

See: New Proxmox Storage Plugin: TrueNAS ZFS over iSCSI via WebSocket API (TrueNAS 24.10 and 25.05 Compatible)

7

I don’t like to double-post, but that was getting a bit long. I wanted to include a link to this Reddit post to better explain the difference between Proxmox’s “iSCSI” and “ZFS over iSCSI” storage types: https://www.reddit.com/r/Proxmox/comments/t27kgu/comment/hyy348o/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

ZFS over iSCSI is a combination of two technologies. This is what happens exactly:

  • proxmox host ssh’s into storage with root credentials

  • it runs zfs volume manager commands on a specified pool to create a zfs file system on a slice of that pool.

  • it creates a file on that file system.

  • it uses one of 4 supported iSCSI daemons (Comstart, Istgt, Iet, LIO) to export this file as a LUN/disk via iscsi

  • it maps to the created iSCSI target and presents an emulation of raw disk to the VM

  • inside the VM you can format the disk as you wish ZFS/EXT/NTFS

The benefits are:

a) You are able to use ZFS snapshots. They happen via ssh on the “storage” side.

b) Storage can be almost any Linux or FreeBSD host

c) You may even be able to use something like FreeNAS.

d) Its free

The drawbacks are:

a) no built in HA with a single server. You will have to design one yourself for storage and iSCSI.

b) limited to those specific iSCSI implementations. A version change or tool change could break the plugin.

c) Very sparsely used as you discovered

d) Obviously you cant use storage where you are :

  • NOT allowed to ssh into

  • NOT running ZFS on

  • NOT using one of the 4 select iSCSI toolsets

8

Like I said, it’s an integration that puts the storage target under the management umbrella of PVE. It looks to be rather convenient if/when it works.

I’ve been looking at this and I can’t see how there’s anything stopping someone from creating storage on the back-end, exporting it via an iSCSI target, and consuming it with PVE. You have to perform the steps manually but so what?

In an enterprise datacenter with a $250k storage array there’s a pretty good chance you’ll be attaching storage to Debian or RHEL or SuSE Enterprise or VMware ESXi without the benefit of these integrations. In fact it’s often separate teams managing this stuff and having the virtualization control plane digging its fingers into the storage layer is unwelcome.

9

I don’t have enough experience yet to have the answer to that. It think that’s how the ZFS over iSCSI storage type is meant to work, but in this case TrueNAS is the complicating factor.

At the same time, from what I’ve done so far, Proxmox VE Storage Manager (pvesm), which handles interfacing between Proxmox and whatever backing storage you’re using, does some specific, non-standard things with ZFS pools to make them work the way Proxmox VE expects/wants them to work. I do have plenty of experience with that, and a lot of it is very confusingly different from standard ZFS until you realize how/why it works the way it does.

From the info I quoted above, apparently that difference in implementation includes getting snapshots from VMs onto ZFS storage in Proxmox–which may or may not have implications for the way Proxmox Backup Server works, as well. I’m not sure.

That is to say, Proxmox does just enough to its ZFS implementation and toolchain to make it not-quite-standard enough that it doesn’t surprise me that it takes some extra effort to talk to an iSCSI target in such a way that Proxmox can just treat it like a local customized for Proxmox ZFS pool–which is, as I understand it, the entire point of the “ZFS over iSCSI” storage type.

PVESM also allows you to just configure individual iSCSI targets from a storage server where an initiator lives, and attach them as VM disks, but you lose some of Proxmox VE’s features and integrations if you do it that way. And I also suspect that storage type won’t work with TrueNAS either, since TrueNAS and Proxmox VE don’t natively support the same iSCSI target protocols.