background image
HomeRecent PostsDrupalSearchTagsRSSContactAboutAccount
Eric.London's picture

Rsync is a great command line program for copying and sync'ing data. It can use standard SSH protocol (default port 22) to copy files from computer to computer, or locally from one path to another. It frequently comes on linux/unix systems, but if you're using Windoze, I suggest installing Cygwin.

Part One
The first step in this tutorial is to setup passwordless SSH. Open a terminal on the computer you want to copy files from, referred to in this article as "local".

# use the ssh-keygen command to generate a public and private key
# I left the passphrase empty, and used the default path: ~/.ssh/id_dsa
local$ ssh-keygen -t dsa

# the above command will create two files (public and private keys)
local$ ls -l ~/.ssh/id_dsa*
-rw-------  1 Eric  staff  668 Feb 26 11:32 /Users/Eric/.ssh/id_dsa
-rw-r--r--  1 Eric  staff  611 Feb 26 11:32 /Users/Eric/.ssh/id_dsa.pub

SCP the public key file (id_dsa.pub) to the computer that will receive the files, referred to as "remote".

# NOTE: you'll need to replace "Eric@remote" with your remote username and IP address
local$ scp ~/.ssh/id_dsa.pub Eric@remote:~/.ssh/id_dsa.pub.transferred

SSH to the remote system and execute a few commands to enable passwordless SSH

$ SSH to remote system
local$ ssh Eric@remote

# append public key to "authorized_keys"
remote$ cat ~/.ssh/id_dsa.pub.transferred >> ~/.ssh/authorized_keys

# remove obsolete public key
remote$ rm ~/.ssh/id_dsa.pub.transferred

# exit remote system
remote$ exit

To verify that the public/private keys are working, SSH to the remote system. You should not be prompted for a password this time.

Part Two
The second step of this tutorial is creating an executable shell script that will transfer the files. I chose to put my scripts in the folder "~/scripts/", but you could put them anywhere you want.

Open up your favorite text editor (emacs, vi, nano, etc) and enter your rsync command.

#!/bin/bash
rsync -avz --delete /path/on/local/computer/ Eric@remote:/path/on/remote/computer/

Please note, the "--delete" flag is optional, and will remove files on the remote computer that do not exist on the local computer. Please use caution.

For my real life example, I setup a script to rsync my iTunes library from my iMac to my MacBookPro.

#!/bin/bash
rsync -avz --delete --exclude '*.m4v' --exclude '*.mp4' ~/Music/iTunes/ Eric@remote:~/Music/iTunes/

After saving the script, set it to be executable using chmod.

local$ chmod u+x /path/to/local/rsync.script.sh

Test your script on the command line, and then SSH to the remote computer to verify the copied files.

local$ /path/to/local/rsync.script.sh

If all is working well, you can setup a cron job to run at your desired time interval. Remember, both computers must be running for this to be automated, so choose a time you know they'll both be on. For example, to run this script daily..

local$ crontab -e

# min hour dayMonth month dayWeek command
0 0 * * * /path/to/local/rsync.script.sh

Version control is an essential tool when it comes to maintaining your code and properly tracking filesystem changes. In this quick tutorial, I'll show you how to update a Drupal module in a subversion integrated environment using rsync. Since simply copying the contents of a new module update on top of your current directory structure is a bad idea (since it will NOT account for file deletions), rsync is a great solution for syncing module update changes. NOTE: it is a bad idea to update a module in a production environment without proper testing; this code assumes you are working in a development environment.

The first step is to download the latest (and in most cases, stable) package for the module you'd like to update to a directory outside your drupal path. For my Drupal installations, I browse to the Available updates page (admin/reports/updates) and copy the URL from this page. Next I go to my shell, use wget to fetch the package to my home directory, and unpack the file:

$ cd ~
$ mkdir Downloads
$ cd Downloads
$ wget http://ftp.drupal.org/files/projects/xmlsitemap-6.x-2.x-dev.tar.gz
$ tar -xzf xmlsitemap-6.x-2.x-dev.tar.gz

Now you can run the rsync command to apply the filesystem changes to your Drupal module directory. You'll have to update the paths listed below to match your filesystem and Drupal installation.

$ rsync -avCz --delete ~/Downloads/xmlsitemap/ /var/www/vhosts/tdb.erl.dev/httpdocs/sites/all/modules/xmlsitemap/

If executed properly, you can change directory to the module you are trying to update and run an "svn stat" command to see what has been updated. NOTE: the below output is simulated to show common svn status changes.

$ cd /var/www/vhosts/tdb.erl.dev/httpdocs/sites/all/modules/xmlsitemap
$ svn stat
?      xmlsitemap.newfile.php
!      xmlsitemap_taxonomy
M      xmlsitemap.module

At this point, you should see a list of all the files that have changed with this module update. You can now run the update.php script in your development environment and verify the changes are working properly. If everything is working properly, you can commit the module update changes...

# remove files that have been deleted
$ svn stat | grep ^! | awk {'print $2'} | xargs -i svn rm '{}'

# add files that have been added
$ svn stat | grep ^? | awk {'print $2'} | xargs -i svn add '{}'

# commit
$ svn commit -m "upgraded xmlsitemap module to version 6.x-2.x-dev"

Now, you can safely deploy the module changes to your production environment (svn update), run the update.php script, and ensure everything is working properly.

Eric.London's picture

Cygwin is a great tool for making a windows system usable. It supplies you with a linux-like environment (POSIX library) that runs on top of windows. Unfortunately, I still have a windows computer in my household (used only as a fileserver), so I rely on cygwin for basic computer commands (rsync, scp, ssh, etc). Here's a script I wrote to backup my iTunes and Photography directories to an external harddrive. Rsync is great because it can can do incremental secure copies!

rsync -az --delete /cygdrive/d/iTunes /cygdrive/f/
rsync -az --delete /cygdrive/d/Photography /cygdrive/f/

Syndicate content