[Dirvish] Dirvish speed and security
Leopold Palomo Avellaneda
leo at alaxarxa.net
Mon May 26 22:06:18 UTC 2008
A Dilluns 26 Maig 2008, Shawn (Red Mop) va escriure:
> On Monday 26 May 2008 02:20:26 pm Leopold Palomo Avellaneda wrote:
> > Wooooww!!
> >
> > How many questions!!!!
>
> Well, I've some more questions for you, and maybe some answers
good!!!
> Can you post your dirvish config files and a log for the backup in
> question?
yes,
the /etc/dirvish/master.conf
#################################
## Example dirvish master configuration file:
bank:
/opt/backup
exclude:
lost+found/
core
*~
.nfs*
.kde/share/cache/*
.firefox/default/*/Cache/*
.nx/cache-*
Runall:
etc 00:30
quasar 00:30
system 00:30
home 00:30
expire-default: +15 days
expire-rule:
# MIN HR DOM MON DOW STRFTIME_FMT
* * * * 1 +3 months
# * * 1-7 * 1 +1 year
# * * 1-7 1,4,7,10 1
* 10-20 * * * +4 days
# * * * * 2-7 +15 days
#################################
the /opt/backup/home/dirvish/
client: ris
tree: /home
xdev: 1
index: gzip
exclude:
/home/robotica/private_stuff/**
#################################
log:
this was from yesterday night, no ones worked on sunday:
ACTION:
rsync -vrltH --delete -pgo --stats -D --numeric-ids -x --exclude-from=/opt/backup/home/20080526003000/exclude --link-dest=/opt/backup/home/20080525003000/tree /home/ /opt/backup/home/20080526003000/tree
building file list ... done
Number of files: 1159048
Number of files transferred: 0
Total file size: 130788706196 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 25919500
File list generation time: 882.634 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 26888462
Total bytes received: 968966
sent 26888462 bytes received 968966 bytes 1191.74 bytes/sec
total size is 130788706196 speedup is 4694.93
but on saturday,
.....
Number of files: 1159048
Number of files transferred: 628
Total file size: 130788706865 bytes
Total transferred file size: 64743072 bytes
Literal data: 64743072 bytes
Matched data: 0 bytes
File list size: 25919500
File list generation time: 927.970 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 91668289
Total bytes received: 983229
sent 91668289 bytes received 983229 bytes 4360.79 bytes/sec
total size is 130788706865 speedup is 1411.62
> Silly question: "speed-limit:" or --bw-limit isn't set anywhere in your
> configs, is it?
nops
> As I understand it, the directory you are trying to back up is /home, and
> that is of the xfs file system with the default mount options. You are
> trying to back this up to an ext3 filesystem with the default mount options
> on an external hard drive attached to another server.
not exactly. The external hard drive is an icy box with a linux inside that
export one nfs fs. It's this kind of little boxes with an arm and a flash
with a configured system.
> Are you using LVM on the server to be backed up? How much free disk space
> is on that server?
yes, lvm and:
/dev/mapper/DiscDur-home
215G 125G 90G 59% /home
59% used.
> > > > Are you backing up over nfs, or rsync/ssh?
> >
> > nfs
>
> Unless you have a very good reason for doing it this way, you should
> probably backup using rsync/ssh instead of nfs. That way, two computers
> are generating the file list instead of just one. It also is easier on
> your network during list generation time.
yes, you are right, but it was the first time that I configured a dirvish and
and I tried to do something quickly ... :-(
> In other words, your "client:" should be the client's hostname or
> usename at hostname, and "tree:" should be the stuff to be backed up.
yes, you are right :-(
> > > > Are you using ssh (-C) or rsync (-z) compression?
> >
> > ummm,
> > index: gzip but I have not found any other parameter. I'm using the
> > default options of the debian package.
>
> index: gzip is just for the indexes, it's not important here.
>
> I'm not too familiar with Debian, but I believe that it is no in both
> cases. With your link speed, it's not all that important, but it may be
> worth a try to add "rsync-option: -z" I don't think that will help, but it
> may be worth trying.
Ok.
> > > > Do you have the --whole-file or --checksum settings selected?
> >
> > nops
>
> whole-file: (B) and checksum: (B) are clear or unset too then?
I don't understand what is this. I don't have neither of that options.
> > > > How much I/O are the drives containing the file systems in question
> > > > under during the backup period when the backup is not running
> > > > compared to how fast they can go?
> >
> > The drives are in a relax position because no one is working at that
> > time, so no big process or whatever. In the morning with all the users
> > working with their homes via nfs, etc, I can do a scp and it goes much
> > more faster and a cp to the nfs mount.
>
> Ok, so nothing here...
Ok.
> > > > What file systems are you using on the backup server and client?
> >
> > the server is xfs and the client is ext3
>
> Noted. I'm not too familiar with dirvish and xfs.
>
> > > > What settings (size, journal type, mount options, block sizes,
> > > > inodes, etc) are those file systems?
> >
> > /home the default and it was formated using defaults. The external backup
> > disk is ext3 but I don't now the settings because it's an external drive.
>
> Check to see if dir_index is enabled on the ext3 filesystem with "tune2fs
> -L (file system device) |grep features" This can help speed up dirvish on
> both ends. Every ext3 system I make uses the same command line: "mke2fs
> (device) -O has_journal,dir_index" Dirvish vaults get "-b 1024 -i 1024" as
> dirvish makes alot of links, and this is more space efficient. Of course,
> most of the files that change on my servers are small, so YMMV on the -b
> and -i.
I can send you this paramenters but I don't think that are relevant. I can
make a scp or cp to the nfs a lot of fast, but rsync no. The external drive
is a raid 0 disk and the fstab
> Can you paste the relevant lines from "mount" ?
> I wonder of the mount option "noatime" can help here. Might be worth
> trying to narrow the problem. "pre-client: mount -o noatime,remount /home"
> and "post-client: mount -o atime,remount /home" might be useful here. This
> leaves atime on for normal operations, but turns it off for dirvish runs.
in the server the fstab:
/dev/mapper/DiscDur-home /home xfs defaults 0 2
in the external hd:
the mount says:
/dev/md1 on /mnt/md1 type ext3 (usrquota,grpquota)
> What kind of external hard drive? (USB, firewire, eSATA, etc)
above.
> > > > How badly fragmented are those file systems?
> >
> > no idea. Probably a bit because the server was formated and uselly
> > intensively during the last 3 years.
>
> Nothing elegant we can do about that anyway.
Ok
> > > And:
> > >
> > > - What speed is the network you are backing to?
> >
> > Gigabit lan both via gigabit switch.
>
> Nothing here.
ok
> > > - What is the existing load on the network?
> >
> > minimal at that time ... I hope!!
>
> Should be nothing here, but worth checking out.
I think that no ones is doing something during the night ....
> > > - Have you tried measuring the raw capacity of your network in copying
> > > files? e.g. copying 1Mb files across the network repeatedly, what is
> > > the sustained throughput (esp. at the times during which your backups
> > > run)?
> >
> > so fast. I have moved files, isos and ok.
>
> Seems ok here.
ok
> > > - Is there traffic shaping or any other bandwidth limiting technology
> > > on the network that might have an impact?
> >
> > No, it's a little lab with one server, several clients and a overflowed
> > system admin that try to keep all in order!!!
>
> Overworked system admin. Either that's redundant, or holy crap, do you
> sleep?
I sleep sometimes :-(
Thanks for all,
Tomorrow more, I should go to sleep!!
Leo
--
--
Linux User 152692
PGP: 0xF944807E
Catalonia
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part.
Url : http://www.dirvish.org/pipermail/dirvish/attachments/20080527/f5618d21/attachment.bin
More information about the Dirvish
mailing list