[Dirvish] How big of a file can be rsync'ed?

Paul Slootman paul at debian.org
Thu Apr 15 09:16:53 UTC 2010


On Tue 16 Mar 2010, Richard wrote:

> I've seen some google hits and I've seen Paul Slootman discussing the 
> issue a few years back.  So what is the final word on transferring large 
> files?

Make sure you're using a recent rsync on both ends of the transfer,
as there have been improvements on how rsync manages the block size with
large files.

Basically AFAIK the file size limit for rsync depends on the OS.

> The encrypted container is 117gig (122BillionBytes).  The initial file 
> was scp'ed up using the option to limit bandwidth. (It took about four 

rsync also has a bandwidth limiting option, which is available via
dirvish. Search the dirvish.conf manpage for speed-limit.

> I had to use the --checksum option for sure because the date and time 
> doesn't change on the file that contains the filesystem.  Is there some 
> other option that I can give?

Hmm, I'm wondering whether --checksum doesn't cause any extra overhead,
wouldn't it be possible to touch the file to update the modification
time prior to running dirvish? Possibly using a pre-client: directive.


Paul


More information about the Dirvish mailing list