background image
HomeRecent PostsDrupalSearchTagsRSSContactAboutAccount
Eric.London's picture

Just thought I'd share a PHP shell script to scan a file system path, search for all .git/.svn directories recursively, and collect a unique list of all remote repository URLs. It uses "svn info" or "git remote" to get the repository URL path.

<?php
// define a path to scan
$scan_path = '/var/www/vhosts';

// keep track of current working dir
$original_cwd = $_SERVER['PWD'];

// get a list of all .git and .svn directories
$files = trim(`find "$scan_path" -type d | egrep -ir '\.(git|svn)$'`);

// explode on "\n"
$files = explode("\n", $files);

// loop through files
$repo_list = array();
foreach(
$files as $key => $file) {

 
// .git or .svn ?
 
$repo_type = substr($file, -4);
 
$repo_path = substr($file, 0, -4);

  switch (
$repo_type) {
    case
'.svn':

     
// get svn repo root
     
$repo_root = trim(`svn info "$repo_path" | grep ^Repository\ Root | sed 's/Repository Root: //'`);
      if (!
in_array($repo_root, $repo_list)) {
       
$repo_list[] = $repo_root;
      }
      break;

    case
'.git':

     
// change dir
     
chdir($repo_path);
     
     
// get git remote path
     
$repo_root = trim(`git remote -v | grep -ir fetch | awk '{print \$2}' | head -1`);
      if (!
in_array($repo_root, $repo_list)) {
       
$repo_list[] = $repo_root;
      }
      break;
  }

}

// sort
sort($repo_list);

// go back to cwd
chdir($original_cwd);

// output repo list into file
file_put_contents('repo_list.txt', implode("\n", $repo_list) . "\n");
?>

I put this PHP in a file called "scan-for-version-control.php", and run it by typing:

$ php scan-for-version-control.php

It created a file in the current working directory: repo_list.txt containing stuff like..

git@myuser.someversioncontrolhost.com:myuser/somerepo1.git
git@myuser.someversioncontrolhost.com:myuser/somerepo2.git
https://myuser.svn.someversioncontrolhost.com/somerepo1
https://myuser.svn.someversioncontrolhost.com/somerepo2

Eric.London's picture

Here's a quick linux command to show all the packages you have installed (on RPM based systems, like RHEL and Centos) and what repository they came from. NOTE: the "sort" command sorts the packages by name, and the "column" command outputs the list in a neatly formatted tab delimited display in your terminal.

$ rpm -qa --qf '%{NAME} %{VENDOR}\n' | sort | column -t

Sample output:

$ rpm -qa --qf '%{NAME} %{VENDOR}\n' | sort | column -t | head
acl                                  CentOS
acpid                                CentOS
alsa-lib                             CentOS
amtu                                 CentOS
anacron                              CentOS
ant                                  JPackage  Project
antlr                                JPackage  Project
apr                                  CentOS
apr                                  CentOS
apr-util                             CentOS

And if you wanted to see all packages that came from external (non-Centos) repositories:

$ rpm -qa --qf '%{NAME} %{VENDOR}\n' | sort | column -t | grep -iv Centos

Eric.London's picture

Adding a lot of untracked files in a Git working copy can be a pain. When I want move control over "git add *" or "git add .", I sometimes run the following and pipe to xargs (and optionally use grep to filter my selection of files). This allows you conditionally add files based on your grep.

# review file modifications
$ git status
#
# ...snip...
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
# ../all/modules/contrib/examples/ajax_example/ajax_example.test
# ../all/modules/contrib/examples/contextual_links_example/
# ../all/modules/contrib/examples/examples.info
# ../all/modules/contrib/examples/examples.module
# ../all/modules/contrib/examples/form_example/
# ../all/modules/contrib/examples/pager_example/
# ../all/modules/contrib/examples/tablesort_example/
# ../all/modules/contrib/examples/theming_example/

# add all untracked files
$ git status -s | grep ^?? | awk '{print $2}' | xargs -i git add '{}'

For example, if you only wanted to add jpg files..

$ git status -s | grep ^?? | awk '{print $2}' | grep -i \.jpg$ | xargs -i git add '{}'

Eric.London's picture

In this tutorial, I'll show how you can use awk, grep, and sed (my favorite command line tools) to backup and archive your MySQL databases. This can be useful to schedule a cron job, transfer your databases to another server, or any other type of scripting.

First, you'll have to get acquainted with connecting to and dumping your database on the command line. Depending on your user, credentials, and where the databases are located, your command might look something like this. Please note, there is no space between the password and the "-p" flag.

$ mysqldump -u user -pPASSWORD -h hostname database > database.sql

To simplify my example, I'm going to shorten the mysqldump command to the follow.

$ mysqldump database > database.sql

Now that we're MySQL command line pros, I'll break down each command. I'll start by showing all the databases.

Eric-Londons-MacBook-Pro:backup Eric$ mysql --execute="show databases"
+---------------------+
| Database            |
+---------------------+
| customers           |
| db_pics_ericlondon  |
| db_thedrupalblog_d6 |
| drupal              |
| drupal-pics         |
| drupalmusicproject  |
| itunes              |
+---------------------+

Now, I'll "pipe" the output from the previous command into awk to show the first column data.

Eric-Londons-MacBook-Pro:backup Eric$ mysql --execute="show databases" | awk '{print $1}'
Database
customers
db_pics_ericlondon
db_thedrupalblog_d6
drupal
drupal-pics
drupalmusicproject
itunes

And use grep to remove the first line that says "Database".

Eric-Londons-MacBook-Pro:backup Eric$ mysql --execute="show databases" | awk '{print $1}' | grep -iv ^Database$
customers
db_pics_ericlondon
db_thedrupalblog_d6
drupal
drupal-pics
drupalmusicproject
itunes

And use sed to build the mysqldump command. This one is kinda tricky, sorry. As you can see, I also embedded the date command in there to generate today's date in the format: YYYYMMDD.

Eric-Londons-MacBook-Pro:backup Eric$ mysql --execute="show databases" | awk '{print $1}' | grep -iv ^Database$ | sed 's/\(.*\)/mysqldump \1 > \1.'$(date +"%Y%m%d")'.sql/'
mysqldump customers > customers.20100825.sql
mysqldump db_pics_ericlondon > db_pics_ericlondon.20100825.sql
mysqldump db_thedrupalblog_d6 > db_thedrupalblog_d6.20100825.sql
mysqldump drupal > drupal.20100825.sql
mysqldump drupal-pics > drupal-pics.20100825.sql
mysqldump drupalmusicproject > drupalmusicproject.20100825.sql
mysqldump itunes > itunes.20100825.sql

Lastly, if everything looks good, you can pipe the output back to the command line.

Eric-Londons-MacBook-Pro:backup Eric$ mysql --execute="show databases" | awk '{print $1}' | grep -iv ^Database$ | sed 's/\(.*\)/mysqldump \1 > \1.'$(date +"%Y%m%d")'.sql/' | sh

Eric-Londons-MacBook-Pro:backup Eric$ ls -1
customers.20100825.sql
db_pics_ericlondon.20100825.sql
db_thedrupalblog_d6.20100825.sql
drupal-pics.20100825.sql
drupal.20100825.sql
drupalmusicproject.20100825.sql
itunes.20100825.sql

You could even take this one step further and pipe the output through gzip to compress the dumps :)

Eric.London's picture

Drush is a great command line module for administering a Drupal site. It's core features are listed here. Check out the README.txt for usage and installation documentation.

Drush can be used to setup a base Drupal installation with a few quick commands, awesome. One of it's biggest time saving features is downloading all the contributed modules when setting up a new site. Here are a few commands I use to setup a new Drupal site based on feature sets. The following commands should be run in the root directory of your drupal installation.

# Download development modules and themes:
drush dl admin admin_menu coder devel devel_themer reroute_email simpletest views_bulk_operations --destination=sites/all/modules/contrib/ --uri=http://yourhostname.com

# Download common/essential modules:
drush dl cck views date wysiwyg pathauto token captcha recaptcha location emfield link jquery_ui webform htmlpurifier luceneapi vertical_tabs --destination=sites/all/modules/contrib/ --uri=http://yourhostname.com

# Download SEO modules:
drush dl google_analytics xmlsitemap globalredirect site_verify --destination=sites/all/modules/contrib/ --uri=http://yourhostname.com

# Download structural/building modules:
drush dl context features ctools panels --destination=sites/all/modules/contrib/ --uri=http://yourhostname.com

# Download image handling modules:
drush dl imageapi transliteration filefield mimedetect imagefield imagecache imagecache_actions --destination=sites/all/modules/contrib/ --uri=http://yourhostname.com

NOTE: the destination and uri flags on the drush command are optional, but I recommend using them. In recent Drupal development I stopped using sites/default in favor of sites/hostname.com, to ensure that if I ever decide to move the site into a multi-site configuration, I will not have overlapping sites/default/files directories.

You can even enable the downloaded modules from the command line by using the "drush en" command..

drush help en

Syndicate content