[Dirvish] Question about Dirvish and Rsync memory usage

[Brian]

Brian wrote:
> Paul Slootman wrote:
>> On Sat 19 Jan 2008, Brian wrote:
>>
>>> I guess this is more of an rsync question, but does involve Dirvish too.
>> You're right, it is more of an rsync question :)
>>
>>> Dirvish keeps around 15 copies of this system (#2) around, of course 
>>> using Hard links as expected.
>>>
>>> The other Debian NSLU (#2) has a big hard disk, and every once in a 
>>> while I rsync the vaults dirs of the NSLU (#1) to NSLU (#2) so as to 
>>> give me a second copy of the backups.
>> This means you're transferring a list that contains 15 copies of an
>> image (assuming there's only one image on #1). Doing this while
>> preserving all hard links is quite memory-intensive, when using current
>> versions of rsync.
>>
>>> Apart from splitting the vault into smaller parts of the system, any 
>> I'd typically do it by first transferring the first image, and then
>> transferring the second one in a similar way that dirvish does, i.e. by
>> using --link-dest that points to the first image, etc.
> Paul,
> hmmm, I guess I still haven't understood what exactly --link-dest will 
> do for me? I am rsyncing with -alH (stupid as I see that -a also 
> includes -l), so I am telling rsync to preserve Hard-links and it seems 
> to be doing that too, according to the space usage, and doing an ls-l, 
> it is. I guess I need to read some more.
> 
> Cheers Brian
> 
> 
>>> ideas how to reduce the memory overhead? Maybe its just a bad idea to 
>>> use dirvish to backup the complete debian system?
>> You'll always run into this sooner or later. Nothing wrong with
>> backuping a complete debian system.
>>
>> What you could try is the current prerelease version of rsync 3.0.0;
>> currently it's prerelease 8.  I'm using that for a couple of systems,
>> and it's much better in doing large lists. For one, it doesn't wait
>> until the entire list is transferred before beginning with transferring
>> files.  Doe note that you need that version on both ends for the new
>> protocol to work.
>>
>>
>> Paul Slootman
>> _______________________________________________
>> Dirvish mailing list
>> Dirvish at dirvish.org
>> http://www.dirvish.org/mailman/listinfo/dirvish
>>
> 
> _______________________________________________
> Dirvish mailing list
> Dirvish at dirvish.org
> http://www.dirvish.org/mailman/listinfo/dirvish
> 
Hi, just FYI, this is the command I am using to backup each tree in my 
vault. Although I have no paging at all now, this will still take about 
5-6 minutes to complete:

  rsync -avH --size-only /public/backup/slug2-system/20071126-1521 
/mnt/sdb2/backup/slug2-system/20071126-1521

I assume this command is correct? I usually add --delete to it too.

So if all the dirvish backups are there, and I want to use --dest-link, 
what should I use as the destination dir?

This is between two USB connected drives on the same Debian slug.
I issue this command for each dir in my VAULT.

Cheers Brian