Home
익명 04:38

unison not synching files after filesystem upgrade to remote server

unison not synching files after filesystem upgrade to remote server

I am running unison version 2.53.2 (ocaml 4.13.1) on each of two systems. One is Raspbian (Debian GNU/Linux 12 (bookworm)) and the other is Ubuntu (Ubuntu 24.04.4 LTS). It may be important to note that the Ubuntu target filesystem is ZFS and the change involved moving from one ZFS to another.

I have had sync running on the bookworm system for years, and it's been working flawlessly. Now I have moved the location of the data to another disk on the Ubuntu server, and unison seems to ignore changes on either end.

The profile is:

include default.prf

label = Synchronize PlexMedia stuff with nasrefresh

logfile = /usr/local/log/unison-plex-nasrefresh.log

root=ssh://dennis@nasrefresh//Main-HDDs/PlexMedia
root=/svr/PlexMedia

perms=0
# repeat=watch

ignore = Path Recoding
ignore = Path recodeLogs

ignore = Name {MegaSAS.*}
ignore = Name {limit}

default.prf is:

prefer = newer
dumbtty = true
copyonconflict = true
group = false
owner = false
auto = true
batch = true
terse = true
times = true
contactquietly = true
ignore = Name {.Trashes}
ignore = Name {.fseventd}
ignore = Name {.Spotlight-V100}
ignore = Name {MegaSAS.*}
ignore = Name {limit}
ignore = Name {._*}
ignore = Name {.DS_Store}

When I run unison a la:

/usr/bin/unison -batch=true -logfile='/usr/local/log/unison-plex-nasrefresh-2026-07.log' plex-nasrefresh >>'/usr/local/log/unison-plex-nasrefresh-2026-07.log' 2>&1

What appears in the log is:

Unison 2.53.2 (ocaml 4.13.1) log started at 2026-07-16 at  5:10:18

This has been running for hours, and no other entries are in the log. On either system, if I run command "top" or other process-watch commands, I see zero CPU consumption for the process, but it is active (meaning, in the list of processes). Example is processes 329037/329038 here:

dennis      2257       1  0 05:10 ?        00:00:00 /usr/bin/unison -batch=true -repeat=watch -logfile=/usr/local/log/unison-movies-nas-2026-07.log movies-nas
dennis      2258    2257  0 05:10 ?        00:00:00 ssh -l dennis nasrefresh -e none unison -server __new-rpc-mode
dennis      2260       1  0 05:10 ?        00:00:00 /usr/bin/unison -batch=true -repeat=watch -logfile=/usr/local/log/unison-macdownload-nasrefresh-2026-07.log macdownload-nasrefresh
dennis      2261    2260  0 05:10 ?        00:00:00 ssh -l dennis nasrefresh -e none unison -server __new-rpc-mode
dennis      2263       1  0 05:10 ?        00:00:01 /usr/bin/unison -batch=true -repeat=watch -logfile=/usr/local/log/unison-msdn-download-2026-07.log msdn-download
dennis      2265       1  0 05:10 ?        00:00:00 /usr/bin/unison -batch=true -repeat=watch -logfile=/usr/local/log/unison-download-nasrefresh-2026-07.log download-nasrefresh
dennis      2267       1  0 05:10 ?        00:00:00 /usr/bin/unison -batch=true -repeat=watch -logfile=/usr/local/log/unison-3dPrint-velmicro-2026-07.log 3dPrint-velmicro
dennis      2268    2263  0 05:10 ?        00:00:00 ssh -l dennis nasrefresh -e none unison -server __new-rpc-mode
dennis      2269    2265  0 05:10 ?        00:00:00 ssh -l dennis nasrefresh -e none unison -server __new-rpc-mode
dennis      2270    2267  0 05:10 ?        00:00:00 ssh -l dennis velmicro -e none unison -server __new-rpc-mode
dennis      2275       1  0 05:10 ?        00:00:00 /usr/bin/unison -batch=true -repeat=watch -logfile=/usr/local/log/unison-picserver-nas-2026-07.log picserver-nas
dennis      2276    2275  0 05:10 ?        00:00:00 ssh -l dennis nasrefresh -e none unison -server __new-rpc-mode
dennis      2278       1  0 05:10 ?        00:00:00 /usr/bin/unison -batch=true -repeat=watch -logfile=/usr/local/log/unison-PaperPortDocs-nas-2026-07.log PaperPortDocs-nas
dennis      2279    2278  0 05:10 ?        00:00:00 ssh -l dennis nasrefresh -e none unison -server __new-rpc-mode
dennis      2280    2267  0 05:10 ?        00:00:00 /usr/bin/unison-fsmonitor
dennis      2281    2257  0 05:10 ?        00:00:00 /usr/bin/unison-fsmonitor
dennis      2282    2260  0 05:10 ?        00:00:00 /usr/bin/unison-fsmonitor
dennis      2303    2263  0 05:10 ?        00:00:00 /usr/bin/unison-fsmonitor
dennis      2304    2275  0 05:10 ?        00:00:00 /usr/bin/unison-fsmonitor
dennis      2305    2265  0 05:10 ?        00:00:00 /usr/bin/unison-fsmonitor
dennis      2306    2278  0 05:10 ?        00:00:00 /usr/bin/unison-fsmonitor
dennis    329037    2046  0 05:57 pts/1    00:00:00 unison plex-nasrefresh-no-repeat.prf
dennis    329038  329037  0 05:57 pts/1    00:00:00 ssh -l dennis nasrefresh -e none unison -server __new-rpc-mode

The move on the remote system SHOULD be invisible to unison, but this clearly worked on the day of the move, when several files were cleaned up in preparation. Here's a part of that log:

/usr/bin/unison -batch=true -repeat=watch -logfile='/usr/local/log/unison-plex-nasrefresh-2026-07.log' plex-nasrefresh >>'/usr/local/log/unison-plex-nasrefresh-2026-07.log' 2>&1 & echo $!

Unison 2.53.2 (ocaml 4.13.1) log started at 2026-07-15 at  0:01:08

deleted  ---->            DVD/13 (2010).iso-converted
Unison 2.53.2 (ocaml 4.13.1) started propagating changes at 00:41:44.32 on 15 Jul 2026
[BGN] Deleting DVD/13 (2010).iso-converted from /svr/PlexMedia
[BGN] Deleting DVD/13 (2010).iso-converted from /svr/PlexMedia
[END] Deleting DVD/13 (2010).iso-converted
[END] Deleting DVD/13 (2010).iso-converted
Unison 2.53.2 (ocaml 4.13.1) finished propagating changes at 00:41:59.62 on 15 Jul 2026, 15.301 s
Synchronization complete at 00:41:59  (1 item transferred, 0 skipped, 0 failed)
Synchronization complete at 00:41:59  (1 item transferred, 0 skipped, 0 failed)
deleted  ---->            DVD/A Fish Called Wanda (1988).iso-converted
deleted  ---->            DVD/Apocolypse Now Redux (2001).iso-converted

The way the filesystem was upgraded:

  • On Ubuntu: backup /Main-HDDs/PlexMedia
  • On Raspbian: stop unison processes
  • On Ubuntu system:
    • Rename /Main-HDDs/PlexMedia to /Main-HDDs/PlexData
    • Create /Main-HDDs/PlexMedia as new zfs Filesystem
    • Move /Main-HDDs/PlexData/* to /Main-HDDs/PlexMedia
  • On Raspbian: start unison processes

Of course, it is possible that I need to delete the archive files for this process and start over - but that introduuces two other points:

  • How to identify the archive files associted with THIS process (this is an ongoing question for me, but not why we're here)
  • (opinion, assuming there's more I need to do) It is unprofessional at best, to silently refuse to take some action because some other action needs to be taken. The system should identify the problem and fail noisily rather than wait quietly and indefinitely for that action to be taken.

Here is a list of the archives on the Raspbian syystem:

> ls ar* fp*
ar064f83872a7c3794f78ccfb8da9a0619  arb6dc07ba9435aecf0a09875b460d21aa  area6959508b0ee1842861cbbb916e62a0  fp3919d21745f8853f22d53e5523505705  fp58f66aa7bdf1519698babdea685552a7  fpc15b5e8e3e6ea8837b52fb3638362214  fpea6959508b0ee1842861cbbb916e62a0
ar3919d21745f8853f22d53e5523505705  arbd8a776b082094de1cd473034c6a9a06  fp064f83872a7c3794f78ccfb8da9a0619  fp39bc0209cc36cbbb11949459a495ec1e  fp5d905e380962b1accc7a20f7d3096dea  fpd8ec70c024d29e1fcd5d011fd31b4e96  fpf02ea00c572b0fc86778ab3bd5c47771
ar39bc0209cc36cbbb11949459a495ec1e  are29cd560a993ce1779a770d18558178e  fp1397f60e16436bfb234814da4018179f  fp49ec73f25a08268cbabd1f41c0dd3c6d  fpb6dc07ba9435aecf0a09875b460d21aa  fpe29cd560a993ce1779a770d18558178e  fpf8d8155b40012ee41bca525d3f7c04f6
ar4e65d3a887c37c3eb0bfa6c6886de4aa  are4dfaceb1a01ed6ca2587453fc45ed9d  fp2583c0e95bbb5805892fa2cd6db5949b  fp4e65d3a887c37c3eb0bfa6c6886de4aa  fpbd8a776b082094de1cd473034c6a9a06  fpe4dfaceb1a01ed6ca2587453fc45ed9d

How can I further diagnose and correct what's going on here?

The path being replicated here is about 2TB, made up as follows. Both these systems are responsive, the initial scan, back in the day, took less than fours hours as I recall.

> du -hs /svr/PlexMedia/*
7.9G    /svr/PlexMedia/Cocomelon
396G    /svr/PlexMedia/Converted-DVDs
0       /svr/PlexMedia/deletemeOnce
600G    /svr/PlexMedia/Favorites-to-Share
3.1G    /svr/PlexMedia/hdHandbrakeRecoded
4.0K    /svr/PlexMedia/hdProcessed
29G     /svr/PlexMedia/hdrecord
301G    /svr/PlexMedia/HPMICRO1-PLEX
48M     /svr/PlexMedia/IFYL Podcasts
2.7G    /svr/PlexMedia/metadata-backup
677M    /svr/PlexMedia/miniDVRecode
769G    /svr/PlexMedia/Movies
2.8G    /svr/PlexMedia/MusicChristmas
16G     /svr/PlexMedia/MusicPlex
56G     /svr/PlexMedia/MyVideos
55G     /svr/PlexMedia/NewStuff
23G     /svr/PlexMedia/QiuMin
134M    /svr/PlexMedia/recodeLogs
4.0K    /svr/PlexMedia/Recoding
835M    /svr/PlexMedia/RECOVERED_FILES
6.1G    /svr/PlexMedia/Tablo
4.0K    /svr/PlexMedia/test
988M    /svr/PlexMedia/The Rest of the Story
8.0K    /svr/PlexMedia/Thumbs.db
457G    /svr/PlexMedia/TV
4.0K    /svr/PlexMedia/where-to-restore.txt
1.6G    /svr/PlexMedia/Yellowstone

Top Answer/Comment:

Now I have moved the location of the data to another disk on the Ubuntu server

Moving files to a new filesystem changes all of their inode numbers. (Even between ZFS datasets on the same pool, as far as I know, a file can't be moved/renamed directly; only copied into a new file with a new zfs object ID and therefore new inum.)

As far as I know, Unison tracks inode numbers alongside file paths to quickly recognize whether a file has been moved or renamed without a full content comparison.

So with every file showing a new inum, Unison has to fall back to reading the files' full contents on both sides to be able to generate the change list. I don't know whether that's really happening, but you should take a look also at the I/O stats (e.g. iotop, or htop's I/O tab) in addition to CPU stats.

How can I further diagnose and correct what's going on here?

First remove the terse = true. With that option specified you're literally telling Unison to stay quiet about what it's doing. Remove the option, then preferably run Unison interactively (with output to tty).

If the status output doesn't say much, run with -debug all or even -debug verbose.

As last resort, attach strace to one of the processes (strace -y -f -p <pid>) to see what it's doing. Strace will show all system calls, i.e. interactions with the kernel. Use also lsfd -p <pid> or lsof -p <pid> or ls -l /proc/<pid>/fd to see whether it is opening any of the sync files.

How to identify the archive files associted with THIS process (this is an ongoing question for me, but not why we're here)

Run head -3 on each of them. Unison's ar files always have a 3-line text header before the binary data begins, and the second line contains the two sync roots.

$ head -3 ar0a32a926714cd9a39c081c76033bbe75 
Unison archive format 22
Archive for root //ember//home/grawity/Attic synchronizing roots //ember//home/grawity/Attic, //frost//home/grawity/Attic
Written at 2020-01-22 at 14:43:17 - case sensitive mode

The fingerprint files don't have such a header, but they're always named after the same path hash as the archive file (which I think was "md5(local_path+remote_path)" or something very similar to that1).


For curiosity, I dug out the "Unison hashname" script from old backups. The algorithm for generating an ar name from the two roots is:

  1. Canonicalize both roots into the //{hostname}/{path} format, e.g.:

    //rain//home/grawity/Videos/Movies
//snow/F:/Movies

  2. Determine which root is local (based on the system's hostname) and combine into a string formatted as:

    {local};{left}, {right};{version}

    This format results in (seemingly deliberately) the same pair of roots having a distinct hash on each machine. Version was 22 when I wrote this script in 2017; it may have changed since. Example:

    //rain//mnt/hdd/Movies;//rain//mnt/Movies, //snow/F:/Movies;22

  3. Encode the string as UTF-8, then generate an MD5 hash. Format the hash as lowercase hexadecimal.

    text = f"{thisroot};{leftroot}, {rightroot};{version}"
digest = hashlib.md5(text.encode("utf-8")).hexdigest()
filename = f"ar{digest}"

    Additional behavior observed in 2017: If the OS is Windows, trim the hash to 6 digits, so that the final ar* file name is no more than 8 characters not counting the extension.

    This is very legacy behavior and hasn't been necessary for Windows since 1995 – I'm not sure whether current Unison versions do this anymore – but my notes say that the build of Unison for Windows I'd been using in 2017 nevertheless did that.

상단 광고의 [X] 버튼을 누르면 내용이 보입니다