[Dirvish] Combining branches after upgrades

Keith Lofstrom keithl at kl-ic.com
Sun Sep 30 16:17:14 UTC 2007

On Sun, Sep 30, 2007 at 01:59:23AM +0100, Darren Hook wrote:
> Hi Keith,
> On 9/29/07, Keith Lofstrom <keithl at kl-ic.com> wrote:
> > Since version 2.6.4, rsync has allowed multiple --link-dest target
> > directories.  If a file is not found in the first --link-dest target,
> > rsync will search the next --link-dest target, then the next, and so
> Hrmm, I can't comment on your idea but, related in some way...
> I did spot the multiple --link-dest thing (or was it --compare-dest?)
> when skimming through the man recently and wondered - could this also
> be used to kick-start from failed backups?
> That is, to use all failed backups up until the last successful image
> as additional references and once a job is successful, the failed
> snapshots could then be deleted.
> How it would work?
> I guess failures could be added to the .hist file as such, so that
> dirvish could track them and include only the last set of failures as
> --link-dest's.

I like the idea.  I don't think we could easily handle multiple failures,
but I think we could come close to your suggestion.  How about potentially
having THREE --link-dest clauses, like so:

   first:  the most recent successful backup for that branch
   second (if needed):  the most recent attempt at backup for that branch
   third: (if needed):  the most recent backup for that vault

That would agressively hunt for files that made it.  One question is
if failed backups leave any incomplete files with misleadingly correct
metadata - could you check that, please? 

I have never had a failed backup;  the closest I get is missing files,
which are usually signs of insufficient exclusions.  Some directories
and files (such as mail queue files) don't need to be backed up, and
can cause missing file warnings and bad exit status if they are, so I
exclude those.

Again, the second and third --link-dest clauses would be optional, and
the logic would have to be worked out, but a scattering of symbolic
links to successful and failed backups could be the structure on which
this is built.  For safety's sake, I would probably mark the failed
backups in some special way, perhaps by transferring them to a 
"failed" subdirectory, or even tag them for deletion after all the new
files they transferred were incorporated into a subsequent successful

The trick with stuff like this is to make sure it has no deleterious
or unexpected effects on the present way of doing backups.  I would
love to have a better programmer than I work out a framework for 
regression testing (and Perl is very good for this) so we don't mess
up dirvish for existing sites.

I would like to hear from the "dirvish is fine and leave it alone"
users before we get very far down this path.


Keith Lofstrom          keithl at keithl.com         Voice (503)-520-1993
KLIC --- Keith Lofstrom Integrated Circuits --- "Your Ideas in Silicon"
Design Contracting in Bipolar and CMOS - Analog, Digital, and Scan ICs

More information about the Dirvish mailing list