[Dirvish] Copying banks

Bernd Haug haug at berndhaug.net
Wed Jan 20 14:31:19 UTC 2010


What do you use when you need to move whole banks to other hosts (or
other file systems)?

rsync -e ssh -aAHXx /mount-point root at remote:/new-mountpoint is very
slow (due to hard link preservation, I presume).

Just dd'ing is out of the question. (E.g. because, in my case, the new
device is slightly (i.e., a few MiB, but still) smaller.)

Do you know some fast way to copy a bank, preserving hard links? OS is Linux.

What I tried:
• see above
• the BSD dump port - it needs too much temp space (i.e., multi-GiB)
for filesystems with a large directory structure and is also quite
slow

I can imagine that making a faster tool that does not have to search
for other files that link there is possible -- in pythocode:
multilinkers = {}
for file in files:
    if multilinked(file):
        if inode(file) in multilinkers:
            link(multilinkers[inode(file)], newname(file))
        else:
            copy(file, newname(file))
            multilinkers[inode(file)] = newname(file)

This should end up using < 1GiB of VM even on pretty big filesystems,
which should be well worth the overhead for faster sync on modern
servers...

But I'd prefer if I didn't have to write it myself...

Yours, Bernd


More information about the Dirvish mailing list