[Dirvish] Dirvish speed and security

[Shawn (Red Mop)]

On Monday 26 May 2008 02:20:26 pm Leopold Palomo Avellaneda wrote:
> Wooooww!!
>
> How many questions!!!!

Well, I've some more questions for you, and maybe some answers

Can you post your dirvish config files and a log for the backup in question?

Silly question:  "speed-limit:" or --bw-limit isn't set anywhere in your 
configs, is it?

As I understand it, the directory you are trying to back up is /home, and that 
is of the xfs file system with the default mount options.  You are trying to 
back this up to an ext3 filesystem with the default mount options on an  
external hard drive attached to another server.

Are you using LVM on the server to be backed up?  How much free disk space is 
on that server?

> A Dilluns 26 Maig 2008, Eric Mountain va escriure:
> > On Monday 26 May 2008, Shawn (Red Mop) spake thus:
> > > List of questions:
> > >
> > > What kind of link are you backing up over?
>
> I don't understand the question. I have a server (dell server with a raid5
> with the homes of my users) and I'm doing a backup to a external hd via
> network.

I was meaning network speed, but that was answered below.

> > > Are you backing up over nfs, or rsync/ssh?
>
> nfs

Unless you have a very good reason for doing it this way, you should probably 
backup using rsync/ssh instead of nfs.  That way, two computers are 
generating the file list instead of just one.  It also is easier on your 
network during list generation time.

In other words, your "client:" should be the client's hostname or 
usename at hostname, and "tree:" should be the stuff to be backed up.

> > > Are you using ssh (-C) or rsync (-z) compression?
>
> ummm,
> index: gzip but I have not found any other parameter. I'm using the default
> options of the debian package.

index: gzip is just for the indexes, it's not important here.

I'm not too familiar with Debian, but I believe that it is no in both cases.  
With your link speed, it's not all that important, but it may be worth a try 
to add "rsync-option: -z"  I don't think that will help, but it may be worth 
trying.

> > > Do you have the --whole-file or --checksum settings selected?
>
> nops

whole-file: (B) and checksum: (B) are clear or unset too then?

> > > How much I/O are the drives containing the file systems in question
> > > under during the backup period when the backup is not running compared
> > > to how fast they can go?
>
> The drives are in a relax position because no one is working at that time,
> so no big process or whatever. In the morning with all the users working
> with their homes via nfs, etc, I can do a scp and it goes much more faster
> and a cp to the nfs mount.

Ok, so nothing here...

> > > What file systems are you using on the backup server and client?
>
> the server is xfs and the client is ext3

Noted.  I'm not too familiar with dirvish and xfs.

> > > What settings (size, journal type, mount options, block sizes, inodes,
> > > etc) are those file systems?
>
> /home the default and it was formated using defaults. The external backup
> disk is ext3 but I don't now the settings because it's an external drive.

Check to see if dir_index is enabled on the ext3 filesystem with "tune2fs -L 
(file system device) |grep features"  This can help speed up dirvish on both 
ends.  Every ext3 system I make uses the same command line: "mke2fs 
(device) -O has_journal,dir_index"  Dirvish vaults get "-b 1024 -i 1024" as 
dirvish makes alot of links, and this is more space efficient.  Of course, 
most of the files that change on my servers are small, so YMMV on the -b 
and -i.

Can you paste the relevant lines from "mount" ?

I wonder of the mount option "noatime" can help here.  Might be worth trying 
to narrow the problem.  "pre-client: mount -o noatime,remount /home" 
and "post-client: mount -o atime,remount /home" might be useful here.  This 
leaves atime on for normal operations, but turns it off for dirvish runs.

What kind of external hard drive?  (USB, firewire, eSATA, etc)

> > > How badly fragmented are those file systems?
>
> no idea. Probably a bit because the server was formated and uselly
> intensively during the last 3 years.

Nothing elegant we can do about that anyway.

> > And:
> >
> > - What speed is the network you are backing to?
>
> Gigabit lan both via gigabit switch.

Nothing here.

> > - What is the existing load on the network?
>
> minimal at that time ... I hope!!

Should be nothing here, but worth checking out.

> > - Have you tried measuring the raw capacity of your network in copying
> > files? e.g. copying 1Mb files across the network repeatedly, what is the
> > sustained throughput (esp. at the times during which your backups run)?
>
> so fast. I have moved files, isos and ok.

Seems ok here.

> > - Is there traffic shaping or any other bandwidth limiting technology on
> > the network that might have an impact?
>
> No, it's a little lab with one server, several clients and a overflowed
> system admin that try to keep all in order!!!


Overworked system admin.  Either that's redundant, or holy crap, do you sleep?

Shawn