[Dirvish] Copying banks

Shawn Perry redmopml at comcast.net
Wed Jan 20 15:40:07 UTC 2010


tar preserves hardlinks.  Just make sure you "man tar" and preserve as
much as possible such as --numeric-owner,  --preserve, --sparse,
--same-owner, and --atime-preserve.  You can use --verbose to see what
file it's on, or pipe through "pv" before and/or after ssh/netcat to
see transfer rates.  You can also do one on one side and the other on
the other side.

via ssh:
http://meinit.nl/using-tar-and-ssh-to-efficiently-copy-files-preserving-permissions

vian netcat is faster, but not encrypted:
http://compsoc.dur.ac.uk/~djw/tarpipe.html


On Wed, Jan 20, 2010 at 8:08 AM, Paul Slootman <paul at debian.org> wrote:
> On Wed 20 Jan 2010, Bernd Haug wrote:
>
>> What do you use when you need to move whole banks to other hosts (or
>> other file systems)?
>
> I usually just move the latest image, and let time take care of the rest :)
>
>> • the BSD dump port - it needs too much temp space (i.e., multi-GiB)
>> for filesystems with a large directory structure and is also quite
>> slow
>
> Why temp space? You can pipe from dump to dump
>
>> I can imagine that making a faster tool that does not have to search
>> for other files that link there is possible -- in pythocode:
>> multilinkers = {}
>> for file in files:
>>     if multilinked(file):
>>         if inode(file) in multilinkers:
>>             link(multilinkers[inode(file)], newname(file))
>>         else:
>>             copy(file, newname(file))
>>             multilinkers[inode(file)] = newname(file)
>>
>> This should end up using < 1GiB of VM even on pretty big filesystems,
>> which should be well worth the overhead for faster sync on modern
>> servers...
>
> I can't imagine that this would be  faster than just using rsync.
>
> You could copy each image one by one, which might be quicker:
> (untested :-)
>
> cd $oldvault
> previous="name of 1st available image"
> rsync -aH --numeric-ids $previous/ $newvault/$previous/
> for i in *; do
>    if [ "$i" != "$previous" -a "$i" != "dirvish" ]; then # skip 1st one and dirvish dir
>        rsync -aH --numeric-ids --link-dest=../$previous/ $i/ $newvault/$i/
>        previous="$i"
>    fi
> done
>
>
>
> Paul
> _______________________________________________
> Dirvish mailing list
> Dirvish at dirvish.org
> http://www.dirvish.org/mailman/listinfo/dirvish
>


More information about the Dirvish mailing list