background image
HomeRecent PostsDrupalSearchTagsRSSContactAboutAccount

Geospatial Apache Solr searching in Drupal 7 using the Search API module (Ubuntu version)
Eric.London's picture
Submitted by Eric.London on 2/1/2012
Tags:

In this tutorial, I'll share my notes and code I've used to setup geospatial Apache Solr searching in Drupal 7 using the Search API module. For this tutorial I created a minimal Ubuntu server virtual machine. All the commands should be executed as a user with permission to modify files, or prefixed with "sudo".

The first thing I do with a fresh virtual machine is check for package upgrades.

$ apt-get update
$ apt-get upgrade

I find it cumbersome to type in a virtual machine window, so I'll install open-ssh and ssh from my Mac. If you plan to do so, you'll need to find your virtual machine's IP address using ifconfig. For this tutorial I added local DNS (/etc/hosts) to point "drupal7.vm" to my VM's IP.

$ apt-get install openssh-server

Install the LAMP stack. The following packages will install Apache httpd as a dependency.

$ apt-get install php5 php5-cli php5-common php5-curl php5-gd php5-mysql php-pear mysql-server

At this point, browsing to your VM/server's IP address will give you the standard Apache welcome message:
It works!
This is the default web page for this server.
The web server software is running but no content has been added, yet.

Install version control.

$ apt-get install git-core

Create a mysql database for Drupal 7.

$ mysql -u youruser -p
mysql> create database drupal7;
mysql> grant all privileges on drupal7.* to 'drupal7'@'localhost' identified by 'somepassword';
mysql> exit

Install drush via Pear.

$ pear upgrade-all
$ pear channel-discover pear.drush.org
$ pear install drush/drush

Verifying drush is installed.

$ which drush
/usr/bin/drush
$ drush --version
drush version 4.5

Create an Apache vhost directory

$ mkdir -p /var/www/vhosts

Download drupal via drush

$ cd /var/www/vhosts
$ drush dl drupal
# rename folder (as necessary)
$ mv drupal-7.10 drupal7

Integrate drupal file system with git

$ cd drupal7
$ git init
$ git add .
$ git commit -am "initial commit of drupal7"

Install drupal via drush

$ drush site-install standard --db-url=mysql://dbuser:pass@localhost/dbname

Add Apache2 vhost

$ cd /etc/apache2/sites-available
# create new file, called "drupal7" with contents:
<VirtualHost *:80>
  ServerName drupal7.vm
  DocumentRoot /var/www/vhosts/drupal7
  ErrorLog /var/log/apache2/drupal7-error_log
  CustomLog /var/log/apache2/drupal7-access_log combined
  <Directory /var/www/vhosts/drupal7>
    AllowOverride All
  </Directory>
</VirtualHost>

# create symlink
$ cd ../sites-enabled
$ ln -s ../sites-available/drupal7 001-drupal7.conf

# enable apache2 mod_rewrite module
$ a2enmod rewrite

# restart apache2
$ /etc/init.d/apache2 restart

At this point, browsing to your VM/server's hostname should show a Drupal installation.

Part 2, Tomcat/Solr

Installing java jdk and tomcat6

$ apt-get install openjdk-6-jdk tomcat6 tomcat6-admin tomcat6-common tomcat6-user

Browsing to your VM/server's hostname on port 8080 (ex: http://drupal7.vm:8080) will show the generic Tomcat welcome message:
It works !
If you're seeing this page via a web browser, it means you've setup Tomcat successfully. Congratulations!

Installing Solr in Tomcat

$ mkdir ~/downloads
$ cd ~/downloads
# Download the latest stable version of Apache Solr from:
url: http://www.apache.org/dyn/closer.cgi/lucene/solr/
# example:
$ wget http://www.motorlogy.com/apache//lucene/solr/3.5.0/apache-solr-3.5.0.tgz
$ tar -xzf apache-solr-3.5.0.tgz

Copy/rename java war file into Tomcat webapps directory

$ cp ~/downloads/apache-solr-3.5.0/dist/apache-solr-3.5.0.war /var/lib/tomcat6/webapps/solr.war

Note: copying the java war file into the Tomcat webapps folder will create this directory automatically:

/var/lib/tomcat6/webapps/solr

Copy solr files

$ cp -r ~/downloads/apache-solr-3.5.0/example/solr/ /var/lib/tomcat6/solr/

Create Catalina config file to link war file to solr directory

$ cd /etc/tomcat6/Catalina/localhost
# create new file: "solr.xml", with the contents:
<?xml version="1.0" encoding="UTF-8"?>
<Context docBase="/var/lib/tomcat6/webapps/solr.war" debug="0" privileged="true" allowLinking="true" crossContext="true">
<Environment name="solr/home" type="java.lang.String" value="/var/lib/tomcat6/solr" override="true" />
</Context>

Setup Tomcat admin user(s)

# edit file: /etc/tomcat6/tomcat-users.xml, ensure similar contents exist:
<?xml version='1.0' encoding='utf-8'?>
<tomcat-users>
<role rolename="admin"/>
<role rolename="manager"/>
<user username="eric" password="supersecretpassword" roles="admin,manager"/>
</tomcat-users>

Update webapps WEB-INF/web.xml file

# edit file: /var/lib/tomcat6/webapps/solr/WEB-INF/web.xml, update "solr/home" section to reflect solr path:
<env-entry>
  <env-entry-name>solr/home</env-entry-name>
  <env-entry-value>/var/lib/tomcat6/solr</env-entry-value>
  <env-entry-type>java.lang.String</env-entry-type>
</env-entry>

Download search api drupal modules that contain solr xml configuration files, and copy into solr conf directory

$ mkdir -p /var/www/vhosts/drupal7/sites/all/modules/contrib
$ cd /var/www/vhosts/drupal7/sites/all/modules/contrib
$ drush dl search_api search_api_solr
$ cp /var/www/vhosts/drupal7/sites/all/modules/contrib/search_api_solr/solrconfig.xml /var/lib/tomcat6/solr/conf/
$ cp /var/www/vhosts/drupal7/sites/all/modules/contrib/search_api_solr/schema.xml /var/lib/tomcat6/solr/conf/

Reset tomcat permissions, and restart tomcat

$ cd /var/lib
$ chown -R tomcat6.tomcat6 tomcat6
$ /etc/init.d/tomcat6 restart

You should now be able to browse to the solr admin java page.
Example: http://drupal7.vm:8080/solr/admin/
Solr Admin Page

If things aren't working well at this point, check the Tomcat logs and look for SEVERE log entries

/var/log/tomcat6/catalina.out

In addition, the solr java module should be listed in the Tomcat Web Application Manager
Ex URL: http://drupal7.vm:8080/manager/html

Part 3, Drupal code

Getting the solr-php-client library from code.google.com

$ mkdir -p /var/www/vhosts/drupal7/sites/all/libraries
$ cd /var/www/vhosts/drupal7/sites/all/libraries

# URL: http://code.google.com/p/solr-php-client/downloads/list
# File: SolrPhpClient.r60.2011-05-04.tgz
$ wget http://solr-php-client.googlecode.com/files/SolrPhpClient.r60.2011-05-04...
$ tar -xzf SolrPhpClient.r60.2011-05-04.tgz

Downloading and installing contrib drupal modules

$ cd /var/www/vhosts/drupal7
$ drush dl entity views ctools facetapi
$ drush en search_api search_api_views search_api_solr search_api_facetapi entity views views_ui ctools facetapi

(Optionally) I install devel, admin_menu, and disable overlay/toolbar

$ drush dl devel admin_menu
$ drush en devel admin_menu
$ drush dis overlay toolbar

Add the tomcat/solr server to Search API configuration.
- URL: /admin/config/search/search_api
- click on "+ Add Server"
- server name: Solr 3.5.0
- Service class: Solr service
- Solr host: localhost
- Solr port: 8080
- Solr path: /solr
- click Create Server

You should receive some confirmation messages:
The server was successfully created.
The Solr server could be reached (latency: # ms).
If not, ensure tomcat/solr is reachable at the url you specified and the tomcat service is running.

At this point Solr is ready to send/receive data and index content, but there is nothing to index. For this tutorial, I decided to build off of user profiles and store latitude and longitude using the geolocation field module.

$ drush dl geolocation
$ drush en geolocation

Adding some user profile fields:
- URL: /admin/config/people/accounts/fields
- First Name | field_name_first | Text
- Last Name | field_name_last | Text
- Geolocation | field_geolocation | Geolocation | Latitude/Longitude

I then added a bunch of users with latitude/longitude coordinates.
- URL: /admin/people/create
- note: I used Google Geocoding API to fetch the coordinates: http://code.google.com/apis/maps/documentation/geocoding/

Adding the search api index.
- URL: /admin/config/search/search_api
- click "+ Add index"
- Index name: People
- Item type: User
- Server: Solr 3.5.0
- click: Create Index

On the next admin page, you can select which fields to index. For this tutorial, I chose: User ID, Name, Email, URL, First Name, and Last Name. Unfortunately, at the time of writing this, the geolocation lat/lng fields are not exposed to the Entity API. I assume this is a temporary problem, and there are numerous patches in the geolocation issue queue.
@see (for example):
Property Info callback for Entity API - http://drupal.org/node/1366642
Fix for Search API not picking up the entity to index it's fields - http://drupal.org/node/1320564

I copied code directly from the issues queue, made some modifications, and created a custom module to expose the geolocation field data to the entity api module. In addition, I added a new property "lat_lon" that concatenates lat and lng together with a comma. @see: http://wiki.apache.org/solr/SpatialSearch

<?php
/**
* Implements hook_field_info_alter()
*/
function MYMODULE_field_info_alter(&$info) {
  if (isset($info['geolocation_latlng'])) {
    $info['geolocation_latlng']['property_type'] = 'geolocation';
    $info['geolocation_latlng']['property_callbacks'] = array('geolocation_property_info_callback');
  }
}

function geolocation_property_info_callback(&$info, $entity_type, $field, $instance, $field_type) {
  $name = $field['field_name'];
  $property = &$info[$entity_type]['bundles'][$instance['bundle']]['properties'][$name];

  $property['type'] = ($field['cardinality'] != 1) ? 'list<geolocation>' : 'geolocation';
  $property['getter callback'] = 'entity_metadata_field_verbatim_get';
  $property['setter callback'] = 'entity_metadata_field_verbatim_set';
  $property['auto creation'] = 'geolocation_default_values';
  $property['property info'] = geolocation_data_property_info();

  unset($property['query callback']);
}

function geolocation_default_values() {

  return array(
    'lat' => '',
    'lng' => '',
    'lat_sin' => '',
    'last_name' => '',
    'lat_cos' => '',
    'lat_rad' => '',
    'lat_lon' => '',
  );

}

function geolocation_data_property_info($name = NULL) {

  // Build an array of basic property information for the geolocation field.
  $properties = array(
    'lat' => array(
      'label' => t('Latitude'),
    ),
    'lng' => array(
      'label' => t('Longitude'),
    ),
    'lat_sin' => array(
      'label' => t('Sine of Latitude'),
    ),
    'lat_cos' => array(
      'label' => t('Cosine of Latitude'),
    ),
    'lat_rad' => array(
      'label' => t('Radian Latitude'),
    ),
    'lat_lon' => array(
      'label' => t('Latitude,Longitude'),
    ),
  );

  // Add the default values for each of the address field properties.
  foreach ($properties as $key => &$value) {
   
    switch ($key) {
   
      case 'lat_lon':
        $value += array(
          'description' => !empty($name) ? t('!label of field %name', array('!label' => $value['label'], '%name' => $name)) : '',
          'type' => 'text',
          'getter callback' => '_MYMODULE_geolocation_entity_property_verbatim_get',
          'setter callback' => '_MYMODULE_geolocation_entity_property_verbatim_set',
        );
        break;
   
      default:
        $value += array(
          'description' => !empty($name) ? t('!label of field %name', array('!label' => $value['label'], '%name' => $name)) : '',
          'type' => 'text',
          'getter callback' => 'entity_property_verbatim_get',
          'setter callback' => 'entity_property_verbatim_set',
        );
        break;
   
    }

}

return $properties;
}


function _MYMODULE_geolocation_entity_property_verbatim_get($data, array $options, $name, $type, $info) {
  if (is_array($data) && isset($data['lat']) && isset($data['lng'])) {
    return $data['lat'] . ',' . $data['lng'];
  }
  return '';
}

function _MYMODULE_geolocation_entity_property_verbatim_set(&$data, $name, $value, $langcode, $type, $info) {
  // TODO
  return;
}
?>

I added this code to a custom module, renamed function calls (as necessary), and enabled. Update the solr index to add the new fields to the index.
- URL: /admin/config/search/search_api/index/people/fields
- Expand "Add Related Fields"
- Choose Geolocation, click Add fields
The above will expose the following fields now available to the index:
- Geolocation » Latitude
- Geolocation » Longitude
- Geolocation » Sine of Latitude
- Geolocation » Cosine of Latitude
- Geolocation » Radian Latitude
- Geolocation » Latitude,Longitude
Enable "Geolocation » Latitude,Longitude" and save changes.

Index the content.
- URL: /admin/config/search/search_api/index/people/status
- Click: Index now
- note: if you had already indexed the content, you'll probably need to clear it first
In my environment, I got the following confirmation message:
Successfully indexed 7 items.

I find it to be very helpful to verify the xml response from Solr directly after making changes to the index/schema.
The following URL structure will query solr for all results and return all fields:

Ex URL: http://drupal7.vm:8080/solr/select/?q=&fl=*

A sample XML document response.

<doc>
  <str name="f_ss_search_api_language"/>
  <str name="f_ss_url">http://drupal7.vm/user/3</str>
  <str name="id">people-3</str>
  <str name="index_id">people</str>
  <long name="is_uid">3</long>
  <str name="item_id">3</str>
  <arr name="spell">
    <str>nashua</str>
    <str>nashua@example.com</str>
    <str>nashua</str>
    <str>nashua</str>
    <str>42.933692,-72.278141</str>
  </arr>
  <str name="ss_search_api_id">3</str>
  <str name="ss_search_api_language"/>
  <str name="ss_url">http://drupal7.vm/user/3</str>
  <arr name="t_field_geolocation:lat_lon">
    <str>42.933692,-72.278141</str>
  </arr>
  <arr name="t_field_name_first">
    <str>nashua</str>
  </arr>
  <arr name="t_field_name_last">
    <str>nashua</str>
  </arr>
  <arr name="t_mail">
    <str>nashua@example.com</str>
  </arr>
  <arr name="t_name">
    <str>nashua</str>
  </arr>
</doc>

Take note the field name in the following XML, it is used in the next file edit.

<arr name="t_field_geolocation:lat_lon">
  <str>42.933692,-72.278141</str>
</arr>

Update the solr schema.xml configuration and add the geospatial fieldType and field data.

# Edit file: /var/lib/tomcat6/solr/conf/schema.xml
# Just prior to the closing "</types>" tag, I inserted: (around line 287)
    <fieldType name="point" class="solr.PointType" dimension="2" subFieldSuffix="_d"/>
    <fieldType name="location" class="solr.LatLonType" subFieldSuffix="_coordinate"/>
    <fieldtype name="geohash" class="solr.GeoHashField"/>

# And, just after the opening "<fields>" tag, I inserted:
    <field name="t_field_geolocation:lat_lon" type="location" indexed="true" stored="true"/>
    <dynamicField name="*_coordinate"  type="tdouble" indexed="true"  stored="false"/>

Restart Tomcat

$ /etc/init.d/tomcat6 restart

Since the schema and solr data types have been updated, the content will have to be re-indexed.
- URL: /admin/config/search/search_api/index/people/status
- click: Clear index
- click: Index now

Returning to the solr query above will now show updated xml: (note: no longer an array)

<str name="t_field_geolocation:lat_lon">42.933692,-72.278141</str>

Verify the native solr geospatial searching is working using the following query syntax:
URL: http://drupal7.vm:8080/solr/select/?q=&fl=*&fq={!geofilt sfield=t_field_geolocation:lat_lon pt=42.933692,-72.278141 d=100}
By putting a distance parameter of 100 (kilometers) and Nashua NH coordinates, I get 2 results: Nashua and Portsmouth, awesome.

Create a solr integrated view.
- URL: /admin/structure/views/add
- View name: People
- Show: People
- Create a Page [checked]
- Path: people
- Continue & edit
Note: at this point, you have full reign over view configuration. For this tutorial, I set the format to Grid, and added some fields:
- Geolocation: Latitude,Longitude (indexed)
- Indexed User: Email
- Indexed User: First Name
- Indexed User: Last Name
- Indexed User: Name
Save the view when edits are complete.

Browsing to the view will show something like this:
Ex URL: http://drupal7.vm.people
People View

The next chunk of custom code modifies the solr query executed and adds geospatial filtering.
@see: hook_search_api_solr_query_alter(array &$call_args, SearchApiQueryInterface $query)

<?php
function MYMODULE_search_api_solr_query_alter(array &$call_args, SearchApiQueryInterface $query) {

  $lat = 42.933692;
  $lng = -72.278141;
  $distance = 100;

  $call_args['params']['fq'][] = "{!geofilt sfield=t_field_geolocation:lat_lon pt={$lat},{$lng} d={$distance}}";

}
?>

The above code will limit the view's results using the hardcoded coordinates.
People View 2

Clearly, it works but there are loose ends to tie..
- automatically fetch a user's coordinates to store in the geolocation field
- add a search form to the people view page to allow the user to search for a location (instead of hard coded coordinates, blah)
- translate the user's location search input to coordinates using an API

Hopefully, I can find more time to elaborate on this tutorial in the near future! Cheers.

Implementing a Subversion pre-commit hook to integrate with Drupal
Eric.London's picture
Submitted by Eric.London on 5/8/2011
Tags:

In this article, I'll show an example of how to implement a Subversion pre-commit hook to integrate with Drupal. Pre-commit hooks can be used to execute any arbitrary code, such as deployment procedures, archiving databases, etc. For this example, I will show how to check for the creation of a subversion tag and archive the database.

To get started, I created a local subversion repository.

# create folder for subversion repositories
$ mkdir /var/subversion

# create the subversion repository
$ svnadmin create /var/subversion/project

Upon creating a new local svn repository, a hooks directory will be created. Example: /var/subversion/project/hooks

Inside this directory will be a bunch of sample scripts ending in ".tmpl" which contain example hook scripts. Here are the contents of the example pre-commit hook without comments:

$ cat pre-commit.tmpl | egrep -iv "(^#|^$)"
REPOS="$1"
TXN="$2"
SVNLOOK=/usr/bin/svnlook
$SVNLOOK log -t "$TXN" "$REPOS" | \
   grep "[a-zA-Z0-9]" > /dev/null || exit 1
commit-access-control.pl "$REPOS" "$TXN" commit-access-control.cfg || exit 1
exit 0

As noted in the pre-commit.tmpl file, there are 2 arguments being passed to the pre-commit script:

[1] REPOS-PATH   (the path to this repository)
[2] TXN-NAME     (the name of the txn about to be committed)

I created a new file called "pre-commit" and added the following contents:

#!/bin/bash

/var/www/vhosts/project.vm/scripts/svn-pre-commit.php "$1" "$2"

I then made the file executable.

$ chmod ug+w pre-commit

The above script simply passes the arguments to a PHP script contained with the Drupal project.

In my scripts folder (/var/www/vhosts/project.vm/scripts), I created the PHP script "svn-pre-commit.php" with the following contents:

#!/usr/bin/php
<?php

// get args
$repo = $argv[1];

// define path to mysql backups
$mysql_backups_path = '/var/www/vhosts/project.vm/database';

// define path to drupal docroot
$drupal_docroot_path = '/var/www/vhosts/project.vm/htdocs';

// get changed path
// example output:
// A   tags/20110503/
$svn_look = `svnlook changed $repo`;

// define pattern to break apart svnlook changed
$pattern = '/^\s*([A-Za-z])\s*(.*)$/';

// execute preg match
preg_match($pattern, $svn_look, $matches);
$svn_action = $matches[1];
$svn_changed_path = $matches[2];

// check if a tag is being created
if ($svn_action == 'A' && substr($svn_changed_path, 0, 5)=='tags/') {

  // get tag name
  $exploded = explode('/', $svn_changed_path);
  $tag_name = $exploded[1];
 
  // change dir to drupal docroot
  chdir($drupal_docroot_path);

  // backup mysql database using drush
  `/var/www/drush/drush sql-dump > {$mysql_backups_path}/tag_{$tag_name}.sql`;

}

I also made this script executable:

$ chmod ug+w svn-pre-commit.php

Now, assuming that my Drupal site is integrated with the subversion repository, and development is at a point to deploy/create a new tag, I executed the following command to create the subversion tag:

$ svn cp file:///var/subversion/project/trunk file:///var/subversion/project/tags/beta-0.1 -m "creating beta 0.1 tag"

To verify, I entered the directory containing my database dumps to checkout the result:

$ cd /var/www/vhosts/project.vm/database

$ ls -1
tag_beta-0.1.sql

Creating a Centos server installation for Apache, MySQL, Tomcat, PHP, Drupal, and Solr
Eric.London's picture
Submitted by Eric.London on 4/30/2011
Tags:

In this article, I'll show the commands I have been using to set up a fresh Centos server, configured for Apache, MySQL, PHP, Tomcat, Drupal, and Apache Solr. For my article, I used Parallels to create a virtual machine from the Centos 5.6 64bit ISOs I downloaded. To simply this article, all commands are being executed as root, firewall configurations and performance tweaks are not accounted for.

Once the distribution is installed, the first thing I do is upgrade all packages.

$ yum update

Install PHP, Apache, and MySQL

$ yum install php53 php53-gd php53-mbstring php53-mysql php53-xml mysql-server httpd

Set runlevels for Apache and MySQL

$ chkconfig --level 2345 httpd on
$ chkconfig --level 2345 mysqld on

Install subversion. I chose to use subversion for this article because the Drupal 6.x installation works well with svn:externals to fetch the SolrPhpClient library. All subversion commands are connecting to a local subversion repository. If you are using an external server (like Beanstalk), you will have to transpose all commands from using "file://" to "https://".

$ yum install subversion

Add a new local subversion repository (OPTIONAL).

$ mkdir /var/subversion
$ svnadmin create /var/subversion/example.com
$ svn mkdir file:///var/subversion/example.com/trunk -m "added trunk"
$ svn mkdir file:///var/subversion/example.com/branches -m "added branches"
$ svn mkdir file:///var/subversion/example.com/tags -m "added tags"

Download/setup drush

$ cd /var/www
$ wget http://ftp.drupal.org/files/projects/drush-7.x-4.4.tar.gz
$ tar -xzf drush-7.x-4.4.tar.gz
$ ln -s /var/www/drush/drush /usr/local/bin/drush

Create a vhost location on the server for the Drupal installation.

$ mkdir /var/www/vhosts
$ cd /var/www/vhosts
$ drush dl drupal
$ mv drupal-7.0/ example.com

Integrate the Drupal files with subversion

$ cd /var/www/vhosts/example.com
$ svn co file:///var/subversion/example.com/trunk .
$ svn add * .htaccess
$ svn commit -m "downloaded drupal"

Download the Drupal apachesolr module

# make a folder for contrib modules
$ mkdir /var/www/vhosts/example.com/sites/all/modules/contrib
$ cd /var/www/vhosts/example.com/sites/all/modules/contrib

# note: in the below command, you may be prompted to choose which version of the Solr module to install. I choose option 2 for the Supported version
$ drush dl apachesolr

# commit to subversion
$ cd /var/www/vhosts/example.com/sites/all/modules
$ svn add contrib
$ svn commit -m "added contrib folder and apachesolr module"

Setup MySQL

# start mysql
$ /etc/init.d/mysqld start

# set root password
$ /usr/bin/mysqladmin -u root password 'new-password'

# create new database, user, and set permissions
$ mysql --execute="create database db_example"
$ mysql --execute="grant all privileges on db_example.* to 'example-user'@'localhost' identified by 'some_password'"

Setup Apache vhost

$ cd /etc/httpd/conf.d

# create a new file "example.com.conf", with the contents:

NameVirtualHost *:80

<Directory /var/www/vhosts>
  AllowOverride All
</Directory>

<VirtualHost *:80>
  ServerName example.com
  DocumentRoot /var/www/vhosts/example.com
  ErrorLog logs/example.com-error_log
  CustomLog logs/example.com-access_log common
</VirtualHost>

Reset Apache file permissions. NOTE: you will need a more solid/secure configuration for this!

$ cd /var/www
$ chown -R apache.apache drush*
$ chown -R apache.apache vhosts

Start Apache

$ /etc/init.d/httpd start

Install Drupal via drush

$ cd /var/www/vhosts/example.com

# note: you can set your user 1 username, password, email, etc in the following command if desired. type "drush help si" for more install options
$ drush site-install standard --sites-subdir=example.com --db-url=mysqli://example-user:some_password@localhost/db_example

At this point, you should be able to browse to your site and it will be up and running.
Drupal Installed

Now, we move onto Tomcat and Solr!

Installing Tomcat and Java. The default Centos yum repositories provide Tomcat5. I prefer Tomcat6, so there are some extras steps below and a dependency issue I had to resolve.

# added repo file to get tomcat6:
$ cd /etc/yum.repos.d/
$ wget http://www.jpackage.org/jpackage50.repo

# install Java JDK:
$ yum install java-1.6.0-openjdk

# install tomcat6:
$ yum install tomcat6 tomcat6-admin-webapps tomcat6-webapps

# dang, dependency issue... (!)

java-1.4.2-gcj-compat-1.4.2.0-40jpp.115.x86_64 from base has depsolving problems
  --> Missing Dependency: /usr/bin/rebuild-security-providers is needed by package java-1.4.2-gcj-compat-1.4.2.0-40jpp.115.x86_64 (base)
Error: Missing Dependency: /usr/bin/rebuild-security-providers is needed by package java-1.4.2-gcj-compat-1.4.2.0-40jpp.115.x86_64 (base)
You could try using --skip-broken to work around the problem
You could try running: package-cleanup --problems
                        package-cleanup --dupes
                        rpm -Va --nofiles --nodigest
The program package-cleanup is found in the yum-utils package.

# Fixing dependency issue (OPTIONAL):
$ mkdir ~/downloads
$ cd ~/downloads
$ wget http://plone.lucidsolutions.co.nz/linux/centos/images/jpackage-utils-com...
$ rpm -Uvh jpackage-utils-compat-el5-0.0.1-1.noarch.rpm
$ yum install tomcat6 tomcat6-admin-webapps tomcat6-webapps

# setting tomcat runlevels
$ chkconfig --level 2345 tomcat6 on

# starting tomcat
$ /etc/init.d/tomcat6 start

At this point, you should be able to access Tomcat in your browser (http://example.com:8080)
Tomcat homepage

Downloading Solr Java library.

$ cd ~/downloads
# note: you may need to choose a different mirror to download
$ wget http://www.fightrice.com/mirrors/apache//lucene/solr/1.4.1/apache-solr-1...
$ tar -xzf apache-solr-1.4.1.tgz

# copy/rename solr war file into Tomcat webapps directory
$ cp ~/downloads/apache-solr-1.4.1/dist/apache-solr-1.4.1.war /var/lib/tomcat6/webapps/solr.war

# copy solr files
$ cp -r ~/downloads/apache-solr-1.4.1/example/solr/ /var/lib/tomcat6/solr/

Copying the java war file into the Tomcat webapps folder will create this directory automatically:

/var/lib/tomcat6/webapps/solr

Create Catalina config file to link war file to solr directory:

$ cd /etc/tomcat6/Catalina/localhost

# create new file: "solr.xml", with the contents:

<?xml version="1.0" encoding="UTF-8"?>
<Context docBase="/var/lib/tomcat6/webapps/solr.war" debug="0" privileged="true" allowLinking="true" crossContext="true">
<Environment name="solr/home" type="java.lang.String" value="/var/lib/tomcat6/solr" override="true" />
</Context>

Setup Tomcat admin user(s):

# edit file: /etc/tomcat6/tomcat-users.xml, ensure similar contents exist:

<?xml version='1.0' encoding='utf-8'?>
<tomcat-users>
<role rolename="admin"/>
<role rolename="manager"/>
<user username="eric" password="supersecretpassword" roles="admin,manager"/>
</tomcat-users>

Update WEB-INF/web.xml file:

# edit file: /var/lib/tomcat6/webapps/solr/WEB-INF/web.xml, update section to reflect solr path:

<env-entry>
  <env-entry-name>solr/home</env-entry-name>
  <env-entry-value>/var/lib/tomcat6/solr</env-entry-value>
  <env-entry-type>java.lang.String</env-entry-type>
</env-entry>

Copy conf files from Drupal apachesolr module into Tomcat Solr conf directory (overwrite):

$ cp /var/www/vhosts/example.com/sites/all/modules/contrib/apachesolr/protwords.txt /var/lib/tomcat6/solr/conf/
$ cp /var/www/vhosts/example.com/sites/all/modules/contrib/apachesolr/schema.xml /var/lib/tomcat6/solr/conf/
$ cp /var/www/vhosts/example.com/sites/all/modules/contrib/apachesolr/solrconfig.xml /var/lib/tomcat6/solr/conf/

Reset Tomcat permissions/ownership:

$ cd /var/lib
$ chown -R tomcat.tomcat tomcat6/

Restart Tomcat

$ /etc/init.d/tomcat6 restart

At this point, you should be able to access the solr/admin tomcat Page (http://example.com:8080/solr/admin)
Solr Admin

If things are not working well at this point, check the Tomcat logs:

/var/log/tomcat6/catalina.out

And, ensure the solr java module is listed in the Tomcat Web Application Manager: http://example.com:8080/manager/html

If all is well, you can now enable the Drupal apachesolr modules:

$ cd /var/www/vhosts/example.com
$ drush en apachesolr apachesolr_search apachesolr_taxonomy apachesolr_access --uri=example.com

Log into your Drupal site. NOTE: default account (via drush): admin/admin

Edit default Apache Solr Host Settings:
URL: http://example.com/admin/config/search/apachesolr/server/solr/edit
Change url to: http://example.com:8080/solr, and save form.

Go to Drupal search settings page:
URL: http://example.com/admin/config/search/settings
Change the default search mode to "Apache Solr search", and save form.

Now, you are ready to test the indexing and integration.

Add a new piece of content to test indexing.
# Example:
# http://example.com/node/add/article
# title: Test Article
# Body: test test test

Browse to solr index page:
URL: http://example.com/admin/config/search/apachesolr/index
Select: Index queued content, and click on Begin button
You should see a status message like: "1 item processed successfully." and "Number of documents in index: 0 (1 sent but not yet processed)"
A few minutes later, refreshing the index page should show: "Number of documents in index: 1"

Search for "test" to verify Solr results.
URL: http://example.com/search/site/test
Solr Search Results

You can also review search results via solr/admin
URL: http://example.com:8080/solr/admin/
Enter "test" in query string box and click search

Part 2, Multicore Configuration (OPTIONAL)

If you need to run multiple sites off a single Solr Tomcat installation, you can setup multicore..

Copy the multicore xml file into your solr directory:

$ cp ~/downloads/apache-solr-1.4.1/example/multicore/solr.xml /var/lib/tomcat6/solr/

Create a new directory for each multisite in the solr directory:

$ mkdir /var/lib/tomcat6/solr/example.com

Replicate the solr conf directory into the new multisite directory:

cp -r /var/lib/tomcat6/solr/conf /var/lib/tomcat6/solr/example.com/conf/

Update the solr.xml file:

# edit file: /var/lib/tomcat6/solr/solr.xml, added <core> section for each site:
<cores adminPath="/admin/cores">
  <core name="example.com" instanceDir="example.com" />
</cores>

Restart Tomcat

$ /etc/init.d/tomcat6 restart

Now, your new multicore site will be accessible here: http://example.com:8080/solr/example.com/admin/

Programmatically generate all images for each imagecache preset using a Drush PHP script
Eric.London's picture
Submitted by Eric.London on 11/26/2010
Tags:

My Drupal photo gallery implements a total of 9 imagecache presets, but due to a PHP memory_limit cap of 90MB in my previous hosting contract, I frequently encountered issues generating images.

Since I was on a shared hosting plan, PHP was executed as CGI and a service ran on the server to kill PHP scripts that reached the PHP memory_limit. This resulted in the following error, which prevented imagecache from creating missing images and redirecting the user properly.

Premature end of script headers: index.php

After changing my hosting plan and increasing my server resources, I decided to write a PHP script to programmatically generate all the missing images. This script when executed on the shell using Drush, will get a list of every image in the files directory, and make an HTTP request for each imagecache preset URL.

<?php
// store original working directory
define('ORIGINAL_DIRECTORY', getcwd());

// define site http host
define('HTTP_HOST','pics.ericlondon.com');

// get imagecache presets data
$presets_data = imagecache_presets();

// loop through presets data and collect preset names
$presets = array();
foreach ($presets_data as $key => $value) {
  $presets[] = $value['presetname'];
}

// get a list of pictures
chdir(file_directory_path());
// NOTE: the following line uses find, grep, and sed to generate a list of image files.
// It will vary depending on which file extensions to include and which directories to ignore
$command = 'find . -type f | egrep -ir "\.(gif|jpeg|jpg|png)$" | sed "s/^\.\///" | egrep -iv "^(imagecache|imagefield_thumbs)\/"';
$output = `$command`;
$files = explode("\n", trim($output));
chdir(ORIGINAL_DIRECTORY);

// loop through file list
foreach ($files as $file) {

  // loop through each preset
  foreach ($presets as $preset) {
 
    // define url
    $url = 'http://' . HTTP_HOST . '/' . file_directory_path() . '/imagecache/' . $preset . '/' . $file;
   
    // request url
    $http_request_data = drupal_http_request($url);
   
    // log entry
    $log_entry = $http_request_data->code . " " . $http_request_data->status_message . " " . $url;
    file_put_contents('ic_preset_log.txt', $log_entry . "\n", FILE_APPEND);
 
  }

}
?>

I executed this script on the shell using the following drush command:

drush scr generate_presets.php

The script logs every request and can be reviewed afterward..

$ tail ic_preset_log.txt
200 OK http://pics.ericlondon.com/sites/ericlondon.com/files/imagecache/large_l...
200 OK http://pics.ericlondon.com/sites/ericlondon.com/files/imagecache/large_p...
200 OK http://pics.ericlondon.com/sites/ericlondon.com/files/imagecache/medium/...
200 OK http://pics.ericlondon.com/sites/ericlondon.com/files/imagecache/medium_...
200 OK http://pics.ericlondon.com/sites/ericlondon.com/files/imagecache/medium_...
200 OK http://pics.ericlondon.com/sites/ericlondon.com/files/imagecache/thumbna...
200 OK http://pics.ericlondon.com/sites/ericlondon.com/files/imagecache/thumbna...
200 OK http://pics.ericlondon.com/sites/ericlondon.com/files/imagecache/thumbna...
200 OK http://pics.ericlondon.com/sites/ericlondon.com/files/imagecache/large/u...
200 OK http://pics.ericlondon.com/sites/ericlondon.com/files/imagecache/large_landscape/userpics/10020/2007_07_21_003.JPG

Deploying a site with language translation hostnames and resolving hard coded hostname database records
Eric.London's picture
Submitted by Eric.London on 11/19/2010
Tags:

Sometimes there's no escaping hard coded hostnames being stored in your Drupal database. If you have ever setup language translations (i18n) in production, and then deployed the site to a development environment (with a different hostname), you'll know exactly what I am talking about. The first thing you'll notice in your dev site is that all your urls are linking back to the production site, and you may even have a hard time logging in.

If you query the languages table, you'll see the following:

select language, domain
from {languages}

Results:

language	domain
de		http://de.ericlondon.com
en		http://www.ericlondon.com
es		http://es.ericlondon.com
fr		http://fr.ericlondon.com

At first I thought it would make sense to write a module or a hook_update_n() function to resolve this problem when deploying to another environment, but this method seemed rather clunky, repeatedly running the same revision of a module's update script. In the end, I decided to simply write a PHP script to run via Drush. The following code fetches your default language and all records in the language table, loops through them, and updates the hostname records to match your new hostname...

<?php
// ensure this script is being called from Drush ONLY
// ensure HTTP_HOST is being used
if (
  !is_array($_SERVER['argv'])
  || substr($_SERVER['argv'][0], -9) != 'drush.php'
  || substr($_SERVER['SCRIPT_FILENAME'], -9) != 'drush.php'
  || !strlen($_SERVER['HTTP_HOST'])
) {
  header('HTTP/1.1 403 Forbidden');
  die('Access Denied.');
}

$output = "";

// fetch hostname from globals
$http_host = $_SERVER['HTTP_HOST'];

// ensure http host exists
if (!$http_host || !is_string($http_host)) {
  die('Error occurred fetching http host.');
}

// set default http host
$default_http_host = 'www.' . $http_host;

// define default language
$default_language = 'en';

// define SQL to fetch default language
$sql = "
  select *
  from {languages}
  where language = '%s'
";

// execute SQL
$default_language_record = db_fetch_object(db_query($sql, $default_language));

// ensure record exists
if (!is_object($default_language_record)) {
  die('Error occurred fetching default language.');
}

// update default language record
$default_language_record->domain = 'http://' . $default_http_host;

// set new variable for "language_default"
variable_set('language_default', $default_language_record);
$output .= "Setting language_default variable to: " . $default_language_record->domain . "\n";

// fetch language records
$sql = "
  select *
  from {languages}
";
$resource = db_query($sql);
$language_records = array();
while ($row = db_fetch_object($resource)) {
  $language_records[] = $row;
}

// correct language records
foreach ($language_records as $key => $value) {

  // determine language prefix 
  switch ($value->language) {
    case $default_language:
      $prefix = 'www.';
      break;
    default:
      $prefix = $value->language . '.';
      break;
  }

  $new_domain = 'http://' . $prefix . $http_host;

  $sql = "
    update {languages}
    set domain = '%s'
    where language = '%s'
  ";
  db_query($sql, $new_domain, $value->language);

  $output .= "Setting language record for language: \"" . $value->language . "\" to: " . $new_domain . "\n";

}

drupal_flush_all_caches();

$output .= "Executed: drupal_flush_all_caches()\n";

$output .= "DONE.\n";

echo $output;
?>

I dropped this script (named: fix_env.php) in the root of my Drupal site, and then executed it via Drush using the following shell command:

drush --uri=http://tdb.erl.dev scr fix_env.php

The "--uri" flag allows you to specific a hostname for multi-site environments, and the "scr" flag allows you to specify a php script to execute.

NOTE: I'm in the habit of never using sites/default, because it's a real PITA if you ever try to move your site into a multi-site configuration ^_^

After running the script via Drush, my hostname database records have been corrected to match my development environment...

select language, domain
from {languages}

Results:

language	domain
de		http://de.tdb.erl.dev
en		http://tdb.erl.dev
es		http://es.tdb.erl.dev
fr		http://fr.tdb.erl.dev
Syndicate content