background image
HomeRecent PostsDrupalSearchTagsRSSContactAboutAccount
Eric.London's picture

Adding a lot of untracked files in a Git working copy can be a pain. When I want move control over "git add *" or "git add .", I sometimes run the following and pipe to xargs (and optionally use grep to filter my selection of files). This allows you conditionally add files based on your grep.

# review file modifications
$ git status
#
# ...snip...
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
# ../all/modules/contrib/examples/ajax_example/ajax_example.test
# ../all/modules/contrib/examples/contextual_links_example/
# ../all/modules/contrib/examples/examples.info
# ../all/modules/contrib/examples/examples.module
# ../all/modules/contrib/examples/form_example/
# ../all/modules/contrib/examples/pager_example/
# ../all/modules/contrib/examples/tablesort_example/
# ../all/modules/contrib/examples/theming_example/

# add all untracked files
$ git status -s | grep ^?? | awk '{print $2}' | xargs -i git add '{}'

For example, if you only wanted to add jpg files..

$ git status -s | grep ^?? | awk '{print $2}' | grep -i \.jpg$ | xargs -i git add '{}'

Eric.London's picture

In this tutorial, I'll show how you can use awk, grep, and sed (my favorite command line tools) to backup and archive your MySQL databases. This can be useful to schedule a cron job, transfer your databases to another server, or any other type of scripting.

First, you'll have to get acquainted with connecting to and dumping your database on the command line. Depending on your user, credentials, and where the databases are located, your command might look something like this. Please note, there is no space between the password and the "-p" flag.

$ mysqldump -u user -pPASSWORD -h hostname database > database.sql

To simplify my example, I'm going to shorten the mysqldump command to the follow.

$ mysqldump database > database.sql

Now that we're MySQL command line pros, I'll break down each command. I'll start by showing all the databases.

Eric-Londons-MacBook-Pro:backup Eric$ mysql --execute="show databases"
+---------------------+
| Database            |
+---------------------+
| customers           |
| db_pics_ericlondon  |
| db_thedrupalblog_d6 |
| drupal              |
| drupal-pics         |
| drupalmusicproject  |
| itunes              |
+---------------------+

Now, I'll "pipe" the output from the previous command into awk to show the first column data.

Eric-Londons-MacBook-Pro:backup Eric$ mysql --execute="show databases" | awk '{print $1}'
Database
customers
db_pics_ericlondon
db_thedrupalblog_d6
drupal
drupal-pics
drupalmusicproject
itunes

And use grep to remove the first line that says "Database".

Eric-Londons-MacBook-Pro:backup Eric$ mysql --execute="show databases" | awk '{print $1}' | grep -iv ^Database$
customers
db_pics_ericlondon
db_thedrupalblog_d6
drupal
drupal-pics
drupalmusicproject
itunes

And use sed to build the mysqldump command. This one is kinda tricky, sorry. As you can see, I also embedded the date command in there to generate today's date in the format: YYYYMMDD.

Eric-Londons-MacBook-Pro:backup Eric$ mysql --execute="show databases" | awk '{print $1}' | grep -iv ^Database$ | sed 's/\(.*\)/mysqldump \1 > \1.'$(date +"%Y%m%d")'.sql/'
mysqldump customers > customers.20100825.sql
mysqldump db_pics_ericlondon > db_pics_ericlondon.20100825.sql
mysqldump db_thedrupalblog_d6 > db_thedrupalblog_d6.20100825.sql
mysqldump drupal > drupal.20100825.sql
mysqldump drupal-pics > drupal-pics.20100825.sql
mysqldump drupalmusicproject > drupalmusicproject.20100825.sql
mysqldump itunes > itunes.20100825.sql

Lastly, if everything looks good, you can pipe the output back to the command line.

Eric-Londons-MacBook-Pro:backup Eric$ mysql --execute="show databases" | awk '{print $1}' | grep -iv ^Database$ | sed 's/\(.*\)/mysqldump \1 > \1.'$(date +"%Y%m%d")'.sql/' | sh

Eric-Londons-MacBook-Pro:backup Eric$ ls -1
customers.20100825.sql
db_pics_ericlondon.20100825.sql
db_thedrupalblog_d6.20100825.sql
drupal-pics.20100825.sql
drupal.20100825.sql
drupalmusicproject.20100825.sql
itunes.20100825.sql

You could even take this one step further and pipe the output through gzip to compress the dumps :)

Eric.London's picture

Here's a quick script to reset ownership on a directory and then commit all changes (deletions, additions, and modifications) to subversion...

#!/bin/sh

_DIR="/path/to/my/svn/directory"
_DATE=`date +%Y\-%m\-%d\ %H\:%I\:%S`

_USER="Eric"
_GROUP="Eric"

# reset file ownership
find ${_DIR} -exec chown ${_USER}.${_GROUP} {} \;

# add new files
svn stat ${_DIR} | grep ^? | sed 's/^?      /svn add "/' | sed 's/$/"/' | sh

# remove deleted files
svn stat ${_DIR} | grep ^! | sed 's/^!      /svn del "/' | sed 's/$/"/' | sh

# commit modifications
svn commit ${_DIR} -m "Automated Commit: ${_DATE}"

Eric.London's picture

I recently had to deploy some new code to an old production environment. Like a good doobie, I made all my changes in a local checked out copy of the subversion file system. I did not want to break the production environment, so I copied the entire vhost into my home directory on the server. I tried to execute an svn update command, but it terminated with the message: object of the same name already exists. This means that a file was creating in my development environment (later revision) that was also created in the production environment. When I ran a svn stat command, there were too many additions, deletions, and modifications, so I decided to write a PHP script to compare the directory structure and files of the 2 environments. I used this script to analyze the file system and create a deployment plan...

<?php
// define where all my files are
$path_httpdocs = '/my/first/path/httpdocs';
$path_httpdocs_new = '/my/second/path/httpdocs';

// get a list of files from the 1st location, ignoring subversion folders
chdir($path_httpdocs);
$files_httpdocs = `find . | grep -v \.svn | sort`;
$files_httpdocs = explode("\n", $files_httpdocs);

// get a list of files from the 2nd location, ignoring subversion folders
chdir($path_httpdocs_new);
$files_httpdocs_new = `find . | grep -v \.svn | sort`;
$files_httpdocs_new = explode("\n", $files_httpdocs_new);

// check for file list diffs
$diffs = array_diff($files_httpdocs, $files_httpdocs_new);
sort($diffs);
echo
"### Additions to httpdocs:\n";
print_r($diffs);

// loop through files and check if they are additions
foreach ($diffs as $f) {
  if (
file_exists($path_httpdocs . '/' . $f) && !file_exists($path_httpdocs_new . '/' . $f)) {
   
// copy new addition to new folder
   
copy($path_httpdocs . '/' . $f, $path_httpdocs_new .'/' . $f);
  }
}

// check for file list diffs
$diffs = array_diff($files_httpdocs_new, $files_httpdocs);
sort($diffs);
echo
"### Additions to httpdocs_new:\n";
print_r($diffs);

// do not continue if there are file differences in the 2 directories
if (count(array_diff($files_httpdocs, $files_httpdocs_new)) || count(array_diff($files_httpdocs_new, $files_httpdocs))) {
  echo
"### Clean up file differences before continuing...\n";
  die;
}

// loop through files and check file properties
$diffs = array();
foreach (
$files_httpdocs as $f) {
  if (!
is_file($path_httpdocs . '/' . $f)) continue;

 
// do a diff on the files 
 
$command = "diff \"$path_httpdocs/$f\" \"$path_httpdocs_new/$f\"";
 
$t = `$command`;

 
// if the diff command generated output, store it
 
if (strlen($t)) {
   
$diffs[$f] = $t;
  }

}

// print all the file diffs
print_r($diffs);
?>

Eric.London's picture

Here's a command to add all the new files in your current path to subversion:

svn stat | grep ^? | sed 's/?      /svn add "/' | sed 's/$/"/' | sh
svn commit -m "added all my new files"

Syndicate content