[Dirvish] CPU bound or disk I/O bound? (rsh: option respected?)

Jason Boxman jasonb at edseek.com
Sat Jan 8 23:13:19 PST 2005

On Saturday 08 January 2005 22:40, Keith Lofstrom wrote:
> On Sat, Jan 08, 2005 at 04:17:13PM -0500, Jason Boxman wrote:
> > Does setting 'rsh: rsh' work (or using `ssh` with some additional
> > options) for anyone else in /etc/dirvish/master.conf?
> >
> > (It ought to outright fail unless `rsh` is actually configured and
> > installed.)
> "rsh: rsh" added to /etc/dirvish/master.conf works for me.  That
> is, it *tries* to use rsh and fails, because I have rsh service
> turned off.  There is a small chance that you set the "rsh:" option
> in both master.conf and in $VAULT/dirvish/default.conf, in which
> case dirvish will use the latter file for the option.

Fortunately, this was an easy one.  For non-binary options you can specify 
multiple parameters, each on a newline.  I assumed such would be the case 
with the "rsh: " option.  Specifying it in master.conf as a single line works 
as expected.  In a roundabout way the manual actually mentions this.

man dirvish.conf:
  Single value options are specified by lines  of  the  form  option: value.
  Options expecting  list must be specified in a multi-line format as shown
  here where the lines specifying values are indented by any kind of
  whitespace even if only one value is being specified.

> You might try setting "summary: long" in your master.conf file,
> and see what that tells you.
> To speed up transfers when you have mondo network bandwidth, try
> adding "whole-file: 1" to master.conf and see what that does.
> See dirvish.conf(5) for more details.

Yeah, I had thought about that.  This most recent run was a whole vault of new 
files, so the option would have had no effect.

> You should also try using raw rsync to move some files, and rcp,
> as well as native file transfers between disks.  If your target
> is a Linux box, think about experiments with "hdparm" to tune up
> the disks - that can double the speed.  If it turns out that disk
> speed is the issue, you may find yourself going to striped RAID.

Yeah, I was a bit light on the details originally.  It was really time for 

The fileserver is a dual P3 733MHz box on a SuperMicro 370DLE.  The array 
being backed up is a 4 x 120GB RAID 5 (64K stripe) array on a 3Ware Escalade 
7410 controller on a dedicated PCI bus, 64 @ 33MHz.  I haven't run bonnie++ 
or iozone3 on it, but a quick `hdparm -Tt` gives me about a 34MB/s read 
speed.  The Gigabit card is a rather cheap 32-bit ns83820 based card I plan 
on abandoning soon.  (I actually emailed you about Dirvish last March and I 
think I mentioned I was going to try backups over gigabit and this was one of 
the cards I had at the time...)  The card's good for at least 20MB/s.

The backup server is a slot 1 P3 600MHz.  Its array is on a 3Ware Escalade 
7810 with 2 x 300GB in a RAID 0 configuration with a 128K stripe.  It has an 
old Allied gigabit NIC in the shared 32-bit bus along with the 3Ware.  I ran 
bonnie++ on it and it reported:

Version1.03------Sequential Output------ --Sequential Input- --Random-
                -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine    Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
sarah 2000M:64k  9311  97 84936  81 29314  35 10409  97 60936  31 155.6   6
sarah 2000M:64k  8951  97 84840  81 29563  36 10509  99 62812  25 155.7   7
sarah 2000M:64k  9438  98 84899  81 29923  36 10502  99 62822  24 154.3   6
sarah 2000M:64k  9430  97 85009  81 29618  36 10501  99 62803  25 154.3   7
sarah 2000M:64k  9426  97 84845  81 29728  36 10502  99 62815  24 155.4   6

Both filesystems are XFS.

> All that said, it would be a great addition to the wiki if you
> did some careful experiments and wrote up your results.  For those
> of us poking along with 100BT, 11MB/s sounds pretty good.

It seems I did a lot better this time, once I had it running over `rsh`.  I 
imagine the P3 can handle using `ssh` with arcfour or blowfish without much 
change in these numbers.

When I transferred around 20GB of files mostly < 1MB I ended up with the 

Total bytes received: 22770511126

sent 2125814 bytes  received 22770511126 bytes  8821474.70 bytes/sec
total size is 22761591080  speedup is 1.00

When I copied 200GB of files > 100MB I ended up with just shy of 12MB/s 

Am I looking at the latency of the disk heads repositioning themselves for 


Jason Boxman
Perl Programmer / *NIX Systems Administrator
Shimberg Center for Affordable Housing | University of Florida
http://edseek.com/ - Linux and FOSS stuff

More information about the Dirvish mailing list