Hi,
I’m currently running OPNsense using UFS on a PC Engines APU2E4 (AMD GX-412TC CPU / 4 GB DRAM) with 16gb M.2 SATA storage onboard.
OPNsense being FreeBSD based.
Back when I started UFS was the only option. Right now ZFS is an option, it’s the prime citizen when it comes to filesystems in FreeBSD 
So my question would be: besides the obvious benefits that ZFS brings, will this even perform decently on this light hardware? I feel like I should move the device over the ZFS, but … well, 4GB is only 4GB, and while the CPU has 4 cores/threads, it’s still… eh, not fast 
I’ve seen a lot of talking about performance penalty of ZFS, but personally I haven’t run into any issues there.
My backup box is an Intel N3150, twice as fast as your system, but when I syncoid my data over Tailscale at around 6 MB/s it gets to a maximum of around 40% of CPU usage, but that’s mostly because of Tailscale. And the backup pool has something like 1% free space.
I would really love to hear about the real performance penalty of ZFS.
It’ll be perfectly fine. ZFS generally won’t be at its best in terms of performance on an extremely low spec box, but you’re building a router–it has essentially zero storage load when not actively booting the system.
Performance wise, you won’t know the difference without breaking out tools to do careful A vs B benchmarking.
2 Likes
I’ve run OpnSense on this exact hardware, with ZFS, for many years. No issue.
I even often put Debian on these boxes, with ZFS, and run all sorts on there. Not fast, but it get’s there in the end.
1 Like
I’m also running OPNSense on an APU2 (with a small 16GB M2 SSD), and it’s been great with ZFS. Absolutely zero difference in performance - because there’s so little IO in the first place. I actually reinstalled from a UFS install and switched to ZFS so I could use boot environments for major upgrades: OPNSense and BSD Boot Environments | Geek Cabinet
Also: it turns out the OPNSense installer is kind of amazing. Before I rebooted off of the USB stick, I made a backup of my config. After rebooting, it gave me the option of importing my existing config and running OPNSense from the USB key while it formatted and reinstalled onto my M2 SSD. If I remember right, it could import the config either from my backup on the other USB key or from the SSD before it was formatted. I had perhaps 5 minutes of internet downtime instead of the 30-60 I expected. Neat!
Interesting on 2 points for me. First is, I do the exact same thing, but I’m getting around 190 MB/s over Tailscale. The second point is my 4 cores hover around 19% (at least what I’ve anecdotally seen) during transfer. But my ram jumps up to about 70% and I’ve got 8 GB in mine.
Granted, I’m running pi-hole on mine, not opnSense.
I don’t know what I’m doing wrong, I cannot get opnSense running locally. For some reason, opnSense gets WAN fine, then everything stops at my switch. 
So, based on @mercenary_sysadmin’s suggestion on 2.5 Admins a couple of years ago, I’m running a TP-Link gateway (well and 4 WAPs) and have the Omada controller software running in a VM. But, I digress.
1 Like