background image
HomeRecent PostsDrupalSearchTagsRSSContactAboutAccount
Eric.London's picture

Rsync is a great command line program for copying and sync'ing data. It can use standard SSH protocol (default port 22) to copy files from computer to computer, or locally from one path to another. It frequently comes on linux/unix systems, but if you're using Windoze, I suggest installing Cygwin.

Part One
The first step in this tutorial is to setup passwordless SSH. Open a terminal on the computer you want to copy files from, referred to in this article as "local".

# use the ssh-keygen command to generate a public and private key
# I left the passphrase empty, and used the default path: ~/.ssh/id_dsa
local$ ssh-keygen -t dsa

# the above command will create two files (public and private keys)
local$ ls -l ~/.ssh/id_dsa*
-rw-------  1 Eric  staff  668 Feb 26 11:32 /Users/Eric/.ssh/id_dsa
-rw-r--r--  1 Eric  staff  611 Feb 26 11:32 /Users/Eric/.ssh/id_dsa.pub

SCP the public key file (id_dsa.pub) to the computer that will receive the files, referred to as "remote".

# NOTE: you'll need to replace "Eric@remote" with your remote username and IP address
local$ scp ~/.ssh/id_dsa.pub Eric@remote:~/.ssh/id_dsa.pub.transferred

SSH to the remote system and execute a few commands to enable passwordless SSH

$ SSH to remote system
local$ ssh Eric@remote

# append public key to "authorized_keys"
remote$ cat ~/.ssh/id_dsa.pub.transferred >> ~/.ssh/authorized_keys

# remove obsolete public key
remote$ rm ~/.ssh/id_dsa.pub.transferred

# exit remote system
remote$ exit

To verify that the public/private keys are working, SSH to the remote system. You should not be prompted for a password this time.

Part Two
The second step of this tutorial is creating an executable shell script that will transfer the files. I chose to put my scripts in the folder "~/scripts/", but you could put them anywhere you want.

Open up your favorite text editor (emacs, vi, nano, etc) and enter your rsync command.

#!/bin/bash
rsync -avz --delete /path/on/local/computer/ Eric@remote:/path/on/remote/computer/

Please note, the "--delete" flag is optional, and will remove files on the remote computer that do not exist on the local computer. Please use caution.

For my real life example, I setup a script to rsync my iTunes library from my iMac to my MacBookPro.

#!/bin/bash
rsync -avz --delete --exclude '*.m4v' --exclude '*.mp4' ~/Music/iTunes/ Eric@remote:~/Music/iTunes/

After saving the script, set it to be executable using chmod.

local$ chmod u+x /path/to/local/rsync.script.sh

Test your script on the command line, and then SSH to the remote computer to verify the copied files.

local$ /path/to/local/rsync.script.sh

If all is working well, you can setup a cron job to run at your desired time interval. Remember, both computers must be running for this to be automated, so choose a time you know they'll both be on. For example, to run this script daily..

local$ crontab -e

# min hour dayMonth month dayWeek command
0 0 * * * /path/to/local/rsync.script.sh

I recently got a new MacBook Pro laptop (awesome!) and went through my usual rigmarole of setting up a new [VirtualBox] virtual machine for LAMP development. In previous situations I used bridged network connections which allow my virtual machine to have its own network connection, and acquire an IP address via DHCP. Now that I plan on being more mobile, I was concerned about having a static IP address for my virtual machine (for samba connections, scripting, and any other processes that relies on a static IP address). I decided to explore alternative virtual network connections and changed my virtual NIC to use NAT (the default network connection for VirtualBox). This configuration establishes a virtual NAT for your virtual machines which is great, but the downside is I know have to setup port forwarding to connect to my virtual machine. In bridged network configurations, I could simply SSH or use Samba to connect to my virtual machine by IP address. With NAT I have to setup port forwarding for the services I need to connect to. Initially I decided to setup port forwarding for SSH (port 22) and HTTP (port 80). While my virtual machine was powered down, I executed the following commands. NOTE: you'll have to replace "Centos" with the name of your virtual machine.

$ VBoxManage setextradata "Centos" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/guestssh/Protocol" TCP
$ VBoxManage setextradata "Centos" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/guestssh/GuestPort" 22
$ VBoxManage setextradata "Centos" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/guestssh/HostPort" 2222
$ VBoxManage setextradata "Centos" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/guesthttp/GuestPort" 80
$ VBoxManage setextradata "Centos" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/guesthttp/HostPort" 8080

Now, I can SSH to my virtual machine..

$ ssh -p 2222 Eric@localhost

Or, connect to Apache by browsing to http://localhost:8080

The benefit of this network configuration allows me to travel anywhere with my laptop, use any type of network connection, and not have to worry about changing the way I connect to my virtual machine for development.

Eric.London's picture

Recently, I had to work on a few Drupal sites and only had FTP access to the webservers. One thing is for certain: FTP is slow and painful. I prefer SSH access so I can interact with Subversion, compress files, dump mysql databases, and transfer files securely. I tried to copy the entire remote docroot to my local development environment (using my FTP client, CyberDuck), and the time estimate to copy all the files individually was ridiculous. There are over 500 files in Drupal core alone, not to mention all the 3rd party modules and uploaded files. I decided to upload a tiny PHP file to execute once to backup the filesystem outside the docroot, so I could copy a single compressed file. Before you attempt something like this, you MUST understand the security risk and vulnerability of exposing site archives and having PHP scripts like this on your server. For instance, if someone was able to get a hold of your settings.php file, they'll have access to your MySQL DNS (connection string). Hopefully, your webserver does not have MySQL and other important services exposed through your firewall, but that is a different topic altogether. I'm already having doubts sharing this PHP snippet. I uploaded the following code to a file in the docroot of Drupal, browsed to the web path once, then promptly removed it from existence. Afterward, I was able to copy the entire filesystem as one file (one transfer), maintain my sanity, and saved myself hours of slow FTP transfers.

<?php
// define a list of valid IPs that can access this file
// yes, I know, this will not prevent spoofers, etc
$validIPs = array(
 
'MY-IPADDRESS',
 
'MY-OTHER-IPADDRESS'
);

// ensure the request is coming from a valid IP address
if (!in_array($_SERVER['REMOTE_ADDR'], $validIPs)) die;

// define a path to the archive to create
// VERY IMPORTANT: you must prefix the file path with "../" to ensure the archive is created outside the docroot path!
// NOTE: you may need to update the file path to work in your hosting situation
$filePath = "../backup.tar.gz";

// ensure the file does not already exist
if (file_exists($filePath)) die;

// define the command to execute to compress the site
// NOTE: you may need to specify the full path to the tar command
$command = "tar -czf $filePath .";

// execute the command to compress the entire site
exec($command);

echo
"done."
?>

After this code is executed, be sure to remove the PHP script and archive!

Syndicate content