Using rsync for replication on OS X
I recently began a quest to replicate my house data with a hosting service so if my house goes up in a fire I still have my data elsewhere. The utility that seems to work the best for this is called rsync, and is available on most flavors of Unix, including Apple’s OS X. Why is it good? Well, rsync talks to the remote host and compares file meta-data like modified times, checksums, etc. to determine if a file needs to be transmitted. It can handle partial file retransmission which is very nice if your dealing with large files. It’s much more efficient than other methods that clumsily send all data over the line to have the destination server determine if the data should be discarded or not. Here’s an example of the command I use (disk1 is a directory with a bunch of directories and files):
superbooky:Sean$ rsync -P -z -a -v -e "ssh -l sshuser" \ --partial --delete-after --max-delete=500 \ /Volumes/disk1 bubba.com:~/disk1
While the command switches look scary, they really aren’t that bad. Here’s the explanation:
-P = Display
-z = Compress the data stream
-a = Archive mode – enables switches -rlptgoD
recurse dirs, copy symlinks, preserve permissions:mod times:group:owner:special files
-v = Increase verbosity
-e “ssh -l sshuser” = Run rsync through ssh for encrypted transfers
–partial = Leave partial files on the destination side. This is to allow rsync to simply continue the transfer if a transmission interuption occurs. If you have large multi-GB files this is invaluable.
–delete-after = Delete any files on the destination that were deleted locally after all files have been replicated rather that during the replication.
–max-delete=500 = If rsync starts it’s deletion and goes more than 50 files, exit with an error. This is a safegaurd in case something weird happens.
Using windows? Sorry there’s no simple equivalent that is just available. You’ll need to install software to get this type functionality