Best Way to Move Data from Local Monolithic Dataset to Different Local Dataset?

OpenZFS TrueNAS
1

I’ve got a single, giant dataset, EvacData living on my pool. It contains all the data I transferred off my previous, dying, ext4-based NAS.

Since EvacData is a single dataset containing multiple folders whose contents I’d like to move into their own datasets, I don’t think snapshots/replication will avail me here, so I need to do this the hard(er) way. (Unless I’m missing something.)

I’d like to create new top-level datasets on my pool and transfer the contents of those folders into them, and then once everything is properly organized into datasets, delete EvacData.

I’m not yet very used to manipulating ZFS dataset contents on the CLI, so I’m not sure what the safest thing to do is here. Here’s the result I’d like to achieve.

  1. Assumption: I have already created the new datasets, and they have the correct group/owner/permissions assigned.
  2. Transfer contents of, e.g., FolderOne in EvacDataset to NewDataset1.
  3. Make sure that after the transfer, the contents of NewDataset1 have ownership and permissions matching NewDataset1.

I think rsync is the right tool for this, but I’m not sure of the proper combination of switches. Usually, I’d use rsync -a to preserve the metadata and permissions of the files I’m moving, but given that EvacDataset and, e.g., NewDataset1 will have different owner/group/permissions setups, that doesn’t seem the right approach.

I know that if I don’t use the -a flag I’ll lose the original creation dates on the content I’m moving, but if that’s the price to do it right, that’s fine. :slight_smile:

So, do I just rsync the data I want to move without the -a flag, or is there anything I’m missing?

2

The -a flag in rsync just combines a bunch of other flags - from the man page:

archive mode; equals -rlptgoD (no -H,-A,-X)

You probably want something like -rltD I’m guessing, to get recursive, links, times, and specials+devices.

1 Like
3

rsync -hav --progress /path/to/EvacDataset/ /path/to/NewDataset1/

Notice the trailing slashes: those mean to synchronize the contents of EvacDataset, but not the folder itself. Presuming you created new datasets beneath NewDataset1 in the same places and with the same names as the folders you want to replace in EvacDataset, this will transfer all your files, give you good visible info on progress as it goes, and preserve all properties possible.

Even if you don’t want the exact same hierarchy of folders (datasets) inside NewDataset1 as you had beneath EvacDataset, you probably want to start this way, since it gets you a single neat rsync command to move everything from the old location to the new. If you want any of those datasets somewhere different after you’ve synchronized, you can just zfs rename pool/NewDataset1/child pool/SomeOtherDataset/child and the entire thing will instantly move itself to the new location you want for it.

Rsync itself will make sure that all your data is transferred properly. If you’re worried about it… run the exact same command a second time. Should be blindingly obvious when there’s nothing left to actually transfer, and it’s just scanning stuff (there’s a summary at the end that will tell you about it also).

4 Likes
4

Thank you both. :slight_smile:

I should clarify that I was foolish, er, inexperienced, and as ham-fistedly as possible rsync’d the complete folder structure on an ext4-based QNAP (Diana) into a single dataset on my TrueNAS server, over SMB. That dataset has no children and is charitably described as a pile of disorganized stuff.

admin@vectorsigma[~]$ sudo zfs list | grep -i Tank
Tank                                                           1.62T  36.4T   104K  /mnt/Tank
Tank/AndromedaClusterStore                                     49.9G  36.4T   104K  /mnt/Tank/AndromedaClusterStore
Tank/AndromedaClusterStore/ISOs                                48.8G  36.4T  48.8G  /mnt/Tank/AndromedaClusterStore/ISOs
Tank/AndromedaClusterStore/Templates                           1.06G  36.4T  1.06G  /mnt/Tank/AndromedaClusterStore/Templates
Tank/DianaEvac                                                 1.57T  36.4T  1.57T  /mnt/Tank/DianaEvac

So, I really just have one dataset with actual folders in it.

admin@vectorsigma[/mnt/Tank/DianaEvac]$ pwd
/mnt/Tank/DianaEvac
admin@vectorsigma[/mnt/Tank/DianaEvac]$ ls -lah
total 81K
drwxr-x---  9 artemis diana-archivists   10 Jul  8 16:44  .
drwxr-xr-x  4 root    root                4 Aug 19 11:02  ..
-rwxr-x---  1 artemis diana-archivists 8.1K Jul 21 17:36  .DS_Store
drwxr-x--- 10 artemis diana-archivists   11 Jul  4 16:34  BackupArchives
drwxr-x---  5 artemis diana-archivists    6 Jul  9 18:20 'Game Systems'
drwxr-x---  5 artemis diana-archivists    6 Jul  1 11:33 'JTD Unsorted Files and Archives'
drwxr-x---  4 artemis diana-archivists    5 Jul  1 15:29  Minecraft
drwxr-x--- 10 artemis diana-archivists   11 Jul  1 18:31  VirtualFS
drwxr-x--- 12 artemis diana-archivists   13 Jul  5 11:39  discArchives
drwxr-x---  5 artemis diana-archivists    6 Jul  3 16:14  unorganized

Some of those, like discArchives, BackupArchives, and game Systems, have content that I want to move into their own top level datasets with different owners/groups, perhaps with child datsets as necessary. Some of them, I’m going to have to mount via SMB and look at to figure out how to reorganize them before I move anything. :slight_smile:

But I think I’ve got what I need with that rsync command. I’m really looking forward to getting all this organized properly on the new server. :slight_smile:

5

You’ve probably come to the same conclusion already, but it sounds like you want to make new child datasets as needed and use rsync to copy from the individual directories into the news datasets.

You could do something like:

for directory in dir1 dir2 dir3; do
    zfs create pool/$directory \
    && rsync -avh /original/path/$directory/ /dataset/path/$directory/
done

I’d test that, but something along those lines could help automate this a little.

6

I’ve actually gotten to test this, finally.

On some folders on the original monolithic dataset, rsync works flawlessly. On others, it’s giving me permission errors when I’m signed in as admin, with operation not permitted. These are all strange dotfiles associated with, e.g., iPhoto photo libraries and other invisible Mac files.

rsync: [receiver] mkstemp "/mnt/Tank/BackupArchives/RexCrimBackupArchives/rexCrimMBP14Backup/Pictures/Photos Library.photoslibrary/resources/journals/.Asset.plist.lSjwLJ" failed: Operation not permitted (1)

It only copied about 20 MB of a 10 GB folder.

If I run rsync again it tries to copy the files again (and fails).

I’m not sure what to do at this point. I think I’ll try mounting both the source and destination via SMB on a remote client and using rsync on the remote machine to copy between them. First, I’m going to delete and recreate the target dataset … I just noticed that the owner and group are off, and think it’s probably best to start from scratch.

7

I’m glad this is mostly working. One thing I didn’t take into account is your being logged in as the admin user. I was assuming you’d ssh in as root, which I think would probably solve your permissions errors unless that chunk of the photos library is really a result of the data in motion (files that appear and disappear during an rsync run can give odd errors).

2 Likes
8

The root account is disabled by default on TrueNAS Scale, though I can sudo su into it. I haven’t seen better results running the command as root.

I’m clearing out the ACL/permissions on the offending dataset now. I’m going to recreate them and see where that gets me.

This feels more like a “TrueNAS is obtuse and weird” problem than a ZFS problem, at this point.

1 Like
9

Concur. :slight_smile:

10

I’m getting to the point where, if I had infinite time, I’d teach myself how to run a NAS with bare metal OpenBSD or something. -_-

TrueNAS Scale (BSD-based) never had any of these weird issues for me.

11

Stripping the ACLs and recreating them didn’t help with the rsync issue.

I’m kind of starting to despise TrueNAS, to be honest. This is just one of several little niggling issues I’m dealing with that are keeping me from actually using the storage server.

1 Like
12

@mercenary_sysadmin Okay. I’ve learned my lesson. TrueNAS just does NOT like you trying to do anything complicated on its CLI.

I mounted both the source and destination as an SMB share, using the owner of each dataset as the log-in user for the SMB share, and those mounted.

Then I set up an rsync between them, so that the data has to flow from the NAS down to my client and then back up to the NAS. But this way, it has the correct permissions in each direction, as the owner of source never tries to do anything to destination, and admin on the TrueNAS server isn’t running the sync command.

This appears to be working, though it’s of course much slower than it’d have been if I could have done this on the NAS itself. SMB hates dealing with thousands of tiny files, even via a 10 GbE connection.

13

I haven’t given SCALE a fair shake yet, but I haven’t been especially happy with what I have seen. I’m definitely hanging onto my TryeNAS Core installation as long as I can and will probably be switching to a vanilla FreeBSD or Ubuntu/Debian file server eventually.

14

Have you looked at XigmaNAS?

https://www.xigmanas.org/

15

@tvcvt I much preferred CORE, but this was a new box and I didn’t want to install CORE only to be forced to migrate later, as they’ve already stopped real development for CORE even if they’re not officially EOL’ing it. (The latest version of CORE–the very last official release–stops at BSD 13.3 and rips out the web console because they couldn’t make it work, among other things.

I’d still love to know why I can only do this rsync across SMB shares (though I suspect it would work across NFS or perhaps even via SSH). What’s the point of being admin or especially root if the server’s still going to tell you no in the most oblique, unhelpful way.

More seriously, I suspect this is a combination of the source dataset (or at least the part I’m trying to copy) being a 1:1 duplicate of part of a Macintosh file system and rsync choking on the Mac extended attributes–maybe it’s trying to modify them and can’t? Access times or modification dates or something?

Rsyncing across network shares overcomes that, but nothing is more inefficient and terrible than trying to move a ton of small files over SMB. :stuck_out_tongue:

The good news is if/when I get this all done, I shouldn’t ever have this problem again. This data is just … haunted. :stuck_out_tongue:

@mercenary_sysadmin I’ll check out Xigma, but admittedly I’m not really in a position to redo all my NAS setup right now. Though it’s certainly in the cards when I build the next one. TrueNAS SCALE has not been a fun experience at all.

16

It is astonishingly simple to do things in XigmaNAS that are astonishingly difficult to convince TrueNAS’ UI to do, IME. Not, like, super fancy stuff. I’m talking about the things that should be simple that are just inexplicably beard-gnawing keyboard-smashing frustrating in TrueNAS, and you’re like WHY? WHY? WHY IS THIS SO HARD?

1 Like
17

This.

I’m amazed at the amount of time that I’ve spent trying to figure out how to do relatively simple server admin tasks in TrueNAS. Tasks that were, for all its other faults, either intuitive or well-documented in QNAP’s QTS.

I’m definitely going to consider XigmaNAS for my next build.

EDIT: So, apparently rsync over SSH is one way to get around the permissions issue I’m having.
I’m going to attempt a transfer that way, all on the NAS itself, by ssh’ing into localhost with the user/password of the user that owns the target dataset.

This is still all profoundly stupid, but if I can get that method to work, at least I could write a script to handle it if I ever have to do this again.

18

@mercenary_sysadmin Okay, so, I double-checked over on the Lawrence Systems forums, as I know he uses TrueNAS a lot and manages to make its CLI obey him.

Turns out it was a me and SMB problem, but I’m going to mostly blame SMB. :stuck_out_tongue:
tl;dr: Strip the permissions on the rsync.

There shouldn’t have been anything wrong with using rsync exactly the way I was trying to. So, I went and stared at the man page and actually tried to make myself understand the -a flag a bit better. And one of the things it does is preserve permissions … which, in retrospect, explains exactly why this was choking.

The data originally came off of a Mac OS filesystem (Mac OS permissions!), was synced to an SMB share on an ext4-based QNAP (SMB permissions + ext4-based Linux permissions!) and then rsync’d over SMB from the QNAP to the evac dataset on the TrueNAS (…?! permissions!).

And rsync tends to choke on SMB permissions, from what I’ve read today.

And the error was permission denied on write, so I assume ZFS decided that no, you’re not writing that mish-mash to my beautiful datasets, thank you.

So, I still wanted to preserve as much as I could, in particular any Mac OS hardlinks and as much metadata (dates, Mac OS extended attributes, etc.) as I could. But I couldn’t copy the permissions.

A bit more studying of the man page, and…

rsync -hav -H --no-perms --progress --stats --exclude=.DS_Store

It works.

Number of files: 171,055 (reg: 123,979, dir: 47,076)
Number of created files: 171,055 (reg: 123,979, dir: 47,076)
Number of deleted files: 0
Number of regular files transferred: 123,979
Total file size: 58.25G bytes
Total transferred file size: 58.25G bytes
Literal data: 58.25G bytes
Matched data: 0 bytes
File list size: 9.37M
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 58.28G
Total bytes received: 2.60M

sent 58.28G bytes  received 2.60M bytes  146.99M bytes/sec
total size is 58.25G  speedup is 1.00

ETA: After the rsync works, check your file permissions on the CLI in the target directory.
The files I copied over ended up with the correct group for the target dataset, but still had the (now incorrect) user for the source dataset. a quick sudo chown -R $user:$group $dir fixed that right up. :slight_smile:

ETA 2:
I was trolling the TrueNAS SCALE documentation trying to get a better understanding of the ACL options while setting up a dataset, and accidentally ended up in the Permissions/ACL documentation for CORE, at Permissions | TrueNAS Documentation Hub.

I’m glad I did, as it contains this text on ACL types and their relation to how SMB works, which is not in the documentation for SCALE, as far as I can tell:

Access Control Lists
An Access Control List (ACL) is a set of account permissions associated with a dataset and applied to directories or files within that dataset. ACLs are typically used to manage user interactions with shared datasets and are created when a dataset is added to a pool.

When creating a dataset, you can choose how the ACL can be modified by selecting an ACL Mode:

  • Passthrough only updates ACL entries (ACEs) that are related to the file or directory mode.
  • Restricted does not allow chmod to make changes to files or directories with a non-trivial ACL. An ACL is trivial if it can be fully expressed as a file mode without losing any access rules. Setting the ACL Mode to Restricted is typically used to optimize a dataset for SMB sharing, but can require further optimizations. For example, configuring an rsync task with this dataset could require adding --no-perms as an extra option for the task.

Emphasis added to the last sentence there. If I’d found this right off, it would’ve saved me about half a day of troubleshooting (and I probably wouldn’t have posted this thread).

This was the exact issue I was experiencing with rsync on a share configured with the SMB preset (which is the only one that uses the Restricted ACL type): rsync failed until --no-perms was included in the command.

Now I’m a bit paranoid about how much useful information is in the CORE documentation that I’m not seeing because I’m using the SCALE docs… I’ll have to start checking them both. I’m not going to assume that they’ll have information parity someday.