I would like a way to perform system updates and configuration changes on a test/staging machine, check everything is ok, then deploy those changes verbatim to a running production machine that has a read-only root dataset. The machine is servicing web requests.
My current idea is to have my debian root filesystem as a read-only mounted clone of an unmounted dataset. That way I should be able to zfs receive changes from the test machine, create a new clone, overlay mount the new clone over the old root filesystem, pause incoming requests at the load balancer, and kexec reboot into the new filesystem. I would also have to configure zfsbootmenu to default to the new clone.
Does that sound like it could work and is sane?