[Dirvish] stop dirvish but not loose transfered data
dhoworth at mrc-lmb.cam.ac.uk
Mon Nov 14 08:10:32 EST 2005
On Sat, 2005-11-12 at 00:05 -0800, Keith Lofstrom wrote:
> On Fri, Nov 11, 2005 at 11:50:46PM -0200, Joel Franco wrote:
> > I've used dirvish a lot time and have a specific need: i wanted to use
> > the full bandwith to make the backup at night but, if the data size is
> > too big to transfer, i want to kill the process at 7am, at which the
> > people start working and need the WAN bandwith to use the web.
> > Thinking about it and, supposing that at certain night a have a data
> > size that i didn't get to completely transfer, i do not want to loose it
> > and start again, but continue at the next night.
> Probably the easiest way to do this is to partition your data into
> smaller vaults and run them sequentially using your own script, not
> dirvish-runall. Let's say you divide things up into 10 smaller
> chunks, each taking about an hour on average. You launch them
> in a circular queue, perhaps doing A B C D E F G the first night,
> H I J A B C D E the second night, F G H I J A B C D E (all) the
> third night because they are light backups, and so forth. Complete
> as many 1 hour segments as you can manage, starting the last one at
> 530am or so.
This all sounds like an excellent idea. I think it can be done with
dirvish-runall instead of writing a new script, since dirvish-runall can
take a --config configfile option. So it should be possible to just
set up the lists ABCDEFG etc in separate 'master' conf files and then
invoke 'dirvish-runall --config ABCDEFG.conf' etc on the correct days
> You can use the dirvish include and exclude rules to divide up your
> data into appropriate-sized groups.
> I don't think you want to interrupt the dirvish/rsync process.
> If rsync is so busy that it cannot complete moving a filesystem,
> that implies that there is a lot of change to that filesystem.
> A 16 hour gap in the middle of the move will make too many files
> change or appear or disappear.
> This has the additional advantage that you can initialize new
> vaults in a reasonable time. A vault that incrementally requires
> 60 minutes to move over a slow link may take many hours - perhaps
> days - to initialize, a much bigger problem for you.
> A perl script, perhaps named "dirvish-rotate", that adds rotation
> features to dirvish-runall would be a useful addition to the dirvish
> capability. Alternatively, you can accomplish something similar by
> using multiple master.conf files called on alternating days from cron.
> I hope that gives you some ideas.
More information about the Dirvish