Sunday 24 August 2014

So.... Yeah... Here's some advice if you're using rsync to transfer large datasets from an iMac Mavericks to a Drobo (or other NAS)

Don't use the built in rsync.

I was only getting 18Mb/s transfer from an attache WD 4Tb My Book.

That's crap to start with as it is USB3 and should be much higher.

I tried BlackMagic and what not and... nope... It's not running USB3.
Switched ports on the back of the iMac to see if it was a positional issue.
Nope.
Cable? Hmm. It's blue and says it's USB3...
In any case this post isn't about WD external drives, USB2/3 or iMacs or Mavericks.

It's about rsync.
And I was getting atrocious copy rates to my Drobos.
18-20Mb/s at best.
Gack!!!

So let's look at the installed version of rsync:

/usr/bin/rsync --version
rsync  version 2.6.9  protocol version 29
Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others.

What?

I have brew installed, so I did a brew install rsync.

/usr/local/bin/rsync --version
rsync  version 3.0.9  protocol version 30
Copyright (C) 1996-2011 by Andrew Tridgell, Wayne Davison, and others.

Better, but the surprise was that when I substituted the brew version for the installed version I went from 18-20Mb/s transfer rate to 40-100Mb/s (depending on the file size and file type of course).

It's not a perfect solution and doesn't explain why the WD 4Tb My Book has such atrocious rates, but I have no time to fiddle with that right now as I have 4Tb to transfer and 40-100 is better than 18-20.

So...
Yeah...
Suggestion: Install the brew version of rsync and use it instead.

Oh... Don't suggest putting a rsyncd service on the Drobos.
That's not the point.

No comments:

Post a Comment