Recently I've been tracking price drops in iTunes apps, so I thought I'd roll my own feed parser in Ruby and share the code. In this blog post I'll demonstrate the following: MongoDB for document database storage, MongoID for the Ruby library, Curb for feed fetching, Nokogiri/Nori for XML parsing, Sinatra for a simple web server, and Google charts for a price spark image.

Installed MongoDB via Homebrew

# install 
brew install mongodb

# started service
mongod

Created a new project Gemfile, file: Gemfile

source 'http://rubygems.org'

gem 'mongoid', '~> 3.0'
gem 'curb'
gem 'nokogiri'
gem 'nori'
gem 'sinatra'
gem 'googlecharts'

Installed gems

bundle

Created a MongoID config file, file: mongoid.yml

development:
  sessions:
    default:
      database: itunes_feeds
      hosts:
        - localhost:27017

Created a mongo include file to define class structure for mongo objects, file: mongo.rb

require 'mongoid'

# load mongo conf
Mongoid.load!('mongoid.yml', :development)

class FeedItem
  include Mongoid::Document
  embeds_many :feedItemPrices
end

class FeedItemPrice
  include Mongoid::Document
  embedded_in :feedItem
end

Created a simple class to fetch, parse, and store the iTunes feed data, file: itunes_feed_fetcher.rb

require 'rubygems'
require 'curb'
require 'nokogiri'
require 'nori'

require './mongo.rb'

class ItunesFeedFetcher

  def initialize
    @feed_count = 300
    @feed_url = "https://itunes.apple.com/us/rss/toppaidapplications/limit=#{@feed_count}/xml"
    @feed_items_new = 0
    @feed_items_updated = 0
  end

  def fetch

    # curl feed url
    curld = Curl::Easy.perform @feed_url

    # convert rss feed xml to hash
    nori = Nori.new(:parser => :nokogiri)
    @feed_data = nori.parse curld.body_str

  end

  def process

    return nil if @feed_data.nil?

    @feed_data['feed']['entry'].each do |entry|

      # get itunes id
      itunes_id = entry['id']

      # check if entry exists in database
      existing = FeedItem.where(itunes_id: itunes_id).first

      if !existing.nil?

        # check if entry has been updated
        if entry['updated'].utc >= existing['updated'].utc
          # todo: update entry details
        end

        # get entry price, minus dollar sign
        entry_price = entry['im:price'].scan(/[0-9.]+/).first

        # create feed item price record
        fip = existing.feedItemPrices.create({created: Time.now, price: entry_price})
        fip.save

        @feed_items_updated += 1

      else

        # get entry price, minus dollar sign
        entry_price = entry['im:price'].scan(/[0-9.]+/).first

        # remove entry price, will be embedded instead
        entry.delete 'im:price'

        # set itunes id to entry
        entry['itunes_id'] = itunes_id

        # create new feed item record
        fi = FeedItem.new entry
        fi.save

        # create feed item price record
        fip = fi.feedItemPrices.create({created: Time.now, price: entry_price})
        fip.save

        @feed_items_new += 1

      end

    end

  end

  def report
    "New: #{@feed_items_new}<br/>Updated: #{@feed_items_updated}"
  end

end

Created a simple sinatra website with 2 urls ("/" and "/fetch"), file: sinatra.rb

#!/usr/bin/env ruby

require 'rubygems'
require 'sinatra'
require 'gchart'

require './mongo.rb'
require './itunes_feed_fetcher.rb'

get '/' do

  output = '<table>'
  FeedItem.each do |fi|
    output += "<tr>"
    output += "<td><img src='#{fi['im:image'][0]}' /></td>"
    output += "<td><b>#{fi['im:name']}</b></td>"
    #output += "<td>#{fi['content']}</td>"

    # collect feed item prices
    prices = fi.feedItemPrices.collect {|fip| fip['price'].to_f}

    # create google chart image url
    chart_url = Gchart.sparkline(:data => prices, :size => '120x40', :line_colors => '0077CC')
    output += "<td><img src='#{chart_url}' /></td>"

    output += "</tr>"
  end
  output += "</table>"
  output

end

get '/fetch' do
  iff = ItunesFeedFetcher.new
  iff.fetch
  iff.process
  iff.report
end

I then started the sinatra app:

./sinatra.rb

Browsing to http://localhost:4567/fetch fetches, parses, and stores the data in mongo. sample output:
New: 0
Updated: 300

And, browsing to http://localhost:4567/ shows the feed items with a price spark image. As you can see I randomized the price spark data to make it more interesting.


Source code on Github

This blog article in an update to a previous post Geospatial Apache Solr searching in Drupal 7 using the Search API module (Ubuntu version). In this article, I will outline a step by step guide on how to integrate Drupal 7 with the Search API modules to implement GeoSpatial searching in a View. I used a clean Ubuntu server (12.04 LTS) installation on a VirtualBox virtual machine.

Initial package installation

# update system packages
apt-get update
apt-get upgrade -y

# install ssh server
apt-get install openssh-server -y

# install apache + php
apt-get install php5 php5-cli php5-curl php5-common php5-gd php5-mysql php-pear -y

# install mysql server
apt-get install mysql-server -y

# install tomcat/java
apt-get install tomcat7 tomcat7-admin openjdk-7-jdk -y

# install git
apt-get install git

# install drush via pear
pear channel-discover pear.drush.org
pear install drush/drush
pear upgrade-all

Setting up MySQL

# add mysql db and user
mysql -p --execute="create database drupal7"
mysql -p --execute="GRANT ALL PRIVILEGES ON drupal7.* TO 'drupal7'@'localhost' IDENTIFIED BY 'lapurd'"

Creating the Apache vhost

# create directory
mkdir -p /var/www/vhosts/drupal7
cd /var/www/vhosts/drupal7

# download drupal via drush
drush dl drupal-7
mv drupal-7.15 html

# create a new git repo
git init
git add .
git commit -am "initial commit"

# install drupal via drush
cd html
drush site-install minimal --db-url=mysql://drupal7:lapurd@localhost/drupal7 --account-mail=drupal@ericlondon.com --account-pass=lapurd

Apache Configuration

# add apache vhost
# new file: /etc/apache2/sites-available/drupal7
<VirtualHost *:80>
	ServerAdmin webmaster@drupal7.dev

	ServerName drupal7.dev

	DocumentRoot /var/www/vhosts/drupal7/html
	<Directory /var/www/vhosts/drupal7/html>
		AllowOverride All
	</Directory>

	ErrorLog ${APACHE_LOG_DIR}/drupal7-error.log

	# Possible values include: debug, info, notice, warn, error, crit, alert, emerg.
	LogLevel warn

	CustomLog ${APACHE_LOG_DIR}/drupal7-access.log combined

</VirtualHost>

# enable apache vhost
cd /etc/apache2/sites-enabled
rm 000-default
ln -s ../sites-available/drupal7 000-drupal7

# enable apache mod_rewrite
a2enmod rewrite

# restart apache
apache2ctl restart

Drupal modules setup

cd /var/www/vhosts/drupal7/html

# core modules
drush en field_ui number options taxonomy list -y

# admin menu
drush dl admin_menu -y
drush en admin_menu -y

# views dependency
drush dl ctools -y
drush en ctools -y

# views
drush dl views -y
drush en views views_ui -y

# entity, search api dependency
drush dl entity -y
drush en entity -y

# facetapi, search api dependency
drush dl facetapi -y
drush en facetapi -y

# search api
drush dl search_api -y
drush en search_api search_api_views search_api_facetapi -y

# search_api_solr
mkdir sites/all/libraries
cd sites/all/libraries
wget http://solr-php-client.googlecode.com/files/SolrPhpClient.r60.2011-05-04.tgz
tar -xzf SolrPhpClient.r60.2011-05-04.tgz
cd ../../..
drush dl search_api_solr -y
drush en search_api_solr -y

# geofield module
drush dl geofield -y
drush en geofield -y

Tomcat & Solr setup

# edit file: /etc/tomcat7/tomcat-users.xml
diff tomcat-users.xml tomcat-users.xml.bak.orig 
29,33c29,35
< 
<   <role rolename="admin"/>
<   <role rolename="manager"/>
<   <user username="eric" password="lapurd" roles="admin,manager"/>
< 
---
> <!--
>   <role rolename="tomcat"/>
>   <role rolename="role1"/>
>   <user username="tomcat" password="tomcat" roles="tomcat"/>
>   <user username="both" password="tomcat" roles="tomcat,role1"/>
>   <user username="role1" password="tomcat" roles="role1"/>
> -->

# Download Solr & copy files
mkdir ~/downloads
cd ~/downloads
wget http://archive.apache.org/dist/lucene/solr/3.5.0/apache-solr-3.5.0.tgz
tar -xzf apache-solr-3.5.0.tgz
cp ~/downloads/apache-solr-3.5.0/dist/apache-solr-3.5.0.war /var/lib/tomcat7/webapps/solr.war
cp -r ~/downloads/apache-solr-3.5.0/example/solr/ /var/lib/tomcat7/solr/

# new file: /etc/tomcat7/Catalina/localhost
<?xml version="1.0" encoding="UTF-8"?>
<Context docBase="/var/lib/tomcat7/webapps/solr.war" debug="0" privileged="true" allowLinking="true" crossContext="true">
<Environment name="solr/home" type="java.lang.String" value="/var/lib/tomcat7/solr" override="true" />
</Context>

# edit file: /var/lib/tomcat7/webapps/solr/WEB-INF/web.xml
Uncomment & change this line:
    <env-entry-value>/put/your/solr/home/here</env-entry-value>
to:
    <env-entry-value>/var/lib/tomcat7/solr</env-entry-value>

# copy files from search_api_solr module to tomcat directory
cp sites/all/modules/search_api_solr/schema.xml /var/lib/tomcat7/solr/conf/
cp sites/all/modules/search_api_solr/solrconfig.xml /var/lib/tomcat7/solr/conf/

# reset tomcat permissions
chown -R tomcat7.tomcat7 /var/lib/tomcat7/solr

# restart tomcat
service tomcat7 restart

# NOTE: admin interface URL
http://localhost:8080/solr/admin/

Drupal configuration

Add content type
- URL: /admin/structure/types/add
  - Name: Place (for example)
  - Save content type

Manage fields on new content type
- URL: /admin/structure/types/manage/place/fields
  - Add new field
    - Label: Geo
    - Machine name: field_geo
    - Field Type: Geofield
    - Widget: Latitude / Longitude
  - Save

Add Solr Server
- URL: /admin/config/search/search_api
- Add Server
  - Enter Server Name
  - Service class: choose Solr service
    - Enter Solr host (localhost), port (8080), path (/solr)
  - Save settings

Add solr index
- URL: /admin/config/search/search_api/add_index
  - Index name: GeoSolr
  - Item type: Node
  - Server: 
- Create index

Select fields to index
- URL: /admin/config/search/search_api/index/geosolr/fields
  - Add related fields
    - Choose "Geo", Add fields
    - Choose "The main body text", Add fields
  - Node ID, Integer
  - Title, Fulltext
  - Geo » LatLong Pair » String
  - The main body text » Text
- Save Changes

Added content
- URL: /node/add/place
  - Added coordinates manually using: http://gmaps-samples.googlecode.com/svn/trunk/geocoder/singlegeocode.html/

Index content
- URL: /admin/config/search/search_api/index/geosolr/status

NOTE: Execute direct Solr query to verify data and fields:
http://localhost:8080/solr/select/?q=&fl=*

Solr configuration changes per GeoSpatial

# changes to schema.xml:
root@ubuntu:/var/lib/tomcat7/solr/conf# diff schema.xml schema.xml.bak.orig 
288,291d287
<     <fieldType name="point" class="solr.PointType" dimension="2" subFieldSuffix="_d"/>
<     <fieldType name="location" class="solr.LatLonType" subFieldSuffix="_coordinate"/>
<     <fieldtype name="geohash" class="solr.GeoHashField"/>
< 
296,299d291
< 
<    <field name="ss_field_geo:latlon" type="location" indexed="true" stored="true"/>
<    <dynamicField name="*_coordinate"  type="tdouble" indexed="true"  stored="false"/>
<

# restart tomcat
service tomcat7 restart

# direct solr query to verify geospatial results (using coordinates from Nashua, NH)
http://localhost:8080/solr/select/?q=&fl=*&fq={!geofilt%20sfield=ss_field_geo:latlon%20pt=42.933692,-72.278141%20d=100}

Add new Drupal view

- URL: /admin/structure/views/add
  - view name: GeoSearch
  - show GeoSolr sorted by Unsorted
  - Create a page
    - path: geosearch
  - save and continue
  - Format: Unformatted list
  - Show: Fields
    - Indexed Node: Node ID (Node ID)
    - Indexed Node: Geo (Geo)
    - Indexed Node: Title (Title)
    - The main body text: Text (indexed) (Text (indexed))
  - Save

Lastly, I added a new module "geosearch" and added a hook_search_api_solr_query_alter() function to add the coordinates. It's up to you how you want to integrate the GeoSpatial integration with your Views and site.

function geosearch_search_api_solr_query_alter(array &$call_args, SearchApiQueryInterface $query) {

  $lat = 42.933692;
  $lng = -72.278141;
  $distance = 100;

  $call_args['params']['fq'][] = "{!geofilt sfield=ss_field_geo:latlon pt={$lat},{$lng} d={$distance}}";

}

In this tutorial, I'll share my notes and code I've used to setup geospatial Apache Solr searching in Drupal 7 using the Search API module. For this tutorial I created a minimal Ubuntu server virtual machine. All the commands should be executed as a user with permission to modify files, or prefixed with "sudo".

The first thing I do with a fresh virtual machine is check for package upgrades.

$ apt-get update
$ apt-get upgrade

I find it cumbersome to type in a virtual machine window, so I'll install open-ssh and ssh from my Mac. If you plan to do so, you'll need to find your virtual machine's IP address using ifconfig. For this tutorial I added local DNS (/etc/hosts) to point "drupal7.vm" to my VM's IP.

$ apt-get install openssh-server

Install the LAMP stack. The following packages will install Apache httpd as a dependency.

$ apt-get install php5 php5-cli php5-common php5-curl php5-gd php5-mysql php-pear mysql-server

At this point, browsing to your VM/server's IP address will give you the standard Apache welcome message:
It works!
This is the default web page for this server.
The web server software is running but no content has been added, yet.

Install version control.

$ apt-get install git-core

Create a mysql database for Drupal 7.

$ mysql -u youruser -p
mysql> create database drupal7;
mysql> grant all privileges on drupal7.* to 'drupal7'@'localhost' identified by 'somepassword';
mysql> exit

Install drush via Pear.

$ pear upgrade-all
$ pear channel-discover pear.drush.org
$ pear install drush/drush

Verifying drush is installed.

$ which drush
/usr/bin/drush
$ drush --version
drush version 4.5

Create an Apache vhost directory

$ mkdir -p /var/www/vhosts

Download drupal via drush

$ cd /var/www/vhosts
$ drush dl drupal
# rename folder (as necessary)
$ mv drupal-7.10 drupal7

Integrate drupal file system with git

$ cd drupal7
$ git init
$ git add .
$ git commit -am "initial commit of drupal7"

Install drupal via drush

$ drush site-install standard --db-url=mysql://dbuser:pass@localhost/dbname

Add Apache2 vhost

$ cd /etc/apache2/sites-available
# create new file, called "drupal7" with contents:
<VirtualHost *:80>
  ServerName drupal7.vm
  DocumentRoot /var/www/vhosts/drupal7
  ErrorLog /var/log/apache2/drupal7-error_log
  CustomLog /var/log/apache2/drupal7-access_log combined
  <Directory /var/www/vhosts/drupal7>
    AllowOverride All
  </Directory>
</VirtualHost>

# create symlink
$ cd ../sites-enabled
$ ln -s ../sites-available/drupal7 001-drupal7.conf

# enable apache2 mod_rewrite module
$ a2enmod rewrite

# restart apache2
$ /etc/init.d/apache2 restart

At this point, browsing to your VM/server's hostname should show a Drupal installation.

Part 2, Tomcat/Solr

Installing java jdk and tomcat6

$ apt-get install openjdk-6-jdk tomcat6 tomcat6-admin tomcat6-common tomcat6-user

Browsing to your VM/server's hostname on port 8080 (ex: http://drupal7.vm:8080) will show the generic Tomcat welcome message:
It works !
If you're seeing this page via a web browser, it means you've setup Tomcat successfully. Congratulations!

Installing Solr in Tomcat

$ mkdir ~/downloads
$ cd ~/downloads
# Download the latest stable version of Apache Solr from:
url: http://www.apache.org/dyn/closer.cgi/lucene/solr/
# example:
$ wget http://www.motorlogy.com/apache//lucene/solr/3.5.0/apache-solr-3.5.0.tgz
$ tar -xzf apache-solr-3.5.0.tgz

Copy/rename java war file into Tomcat webapps directory

$ cp ~/downloads/apache-solr-3.5.0/dist/apache-solr-3.5.0.war /var/lib/tomcat6/webapps/solr.war

Note: copying the java war file into the Tomcat webapps folder will create this directory automatically:

/var/lib/tomcat6/webapps/solr

Copy solr files

$ cp -r ~/downloads/apache-solr-3.5.0/example/solr/ /var/lib/tomcat6/solr/

Create Catalina config file to link war file to solr directory

$ cd /etc/tomcat6/Catalina/localhost
# create new file: "solr.xml", with the contents:
<?xml version="1.0" encoding="UTF-8"?>
<Context docBase="/var/lib/tomcat6/webapps/solr.war" debug="0" privileged="true" allowLinking="true" crossContext="true">
<Environment name="solr/home" type="java.lang.String" value="/var/lib/tomcat6/solr" override="true" />
</Context>

Setup Tomcat admin user(s)

# edit file: /etc/tomcat6/tomcat-users.xml, ensure similar contents exist:
<?xml version='1.0' encoding='utf-8'?>
<tomcat-users>
<role rolename="admin"/>
<role rolename="manager"/>
<user username="eric" password="supersecretpassword" roles="admin,manager"/>
</tomcat-users>

Update webapps WEB-INF/web.xml file

# edit file: /var/lib/tomcat6/webapps/solr/WEB-INF/web.xml, update "solr/home" section to reflect solr path:
<env-entry>
  <env-entry-name>solr/home</env-entry-name>
  <env-entry-value>/var/lib/tomcat6/solr</env-entry-value>
  <env-entry-type>java.lang.String</env-entry-type>
</env-entry>

Download search api drupal modules that contain solr xml configuration files, and copy into solr conf directory

$ mkdir -p /var/www/vhosts/drupal7/sites/all/modules/contrib
$ cd /var/www/vhosts/drupal7/sites/all/modules/contrib
$ drush dl search_api search_api_solr
$ cp /var/www/vhosts/drupal7/sites/all/modules/contrib/search_api_solr/solrconfig.xml /var/lib/tomcat6/solr/conf/
$ cp /var/www/vhosts/drupal7/sites/all/modules/contrib/search_api_solr/schema.xml /var/lib/tomcat6/solr/conf/

Reset tomcat permissions, and restart tomcat

$ cd /var/lib
$ chown -R tomcat6.tomcat6 tomcat6
$ /etc/init.d/tomcat6 restart

You should now be able to browse to the solr admin java page.
Example: http://drupal7.vm:8080/solr/admin/


If things aren't working well at this point, check the Tomcat logs and look for SEVERE log entries

/var/log/tomcat6/catalina.out

In addition, the solr java module should be listed in the Tomcat Web Application Manager
Ex URL: http://drupal7.vm:8080/manager/html

Part 3, Drupal code

Getting the solr-php-client library from code.google.com

$ mkdir -p /var/www/vhosts/drupal7/sites/all/libraries
$ cd /var/www/vhosts/drupal7/sites/all/libraries

# URL: http://code.google.com/p/solr-php-client/downloads/list
# File: SolrPhpClient.r60.2011-05-04.tgz
$ wget http://solr-php-client.googlecode.com/files/SolrPhpClient.r60.2011-05-04.tgz
$ tar -xzf SolrPhpClient.r60.2011-05-04.tgz

Downloading and installing contrib drupal modules

$ cd /var/www/vhosts/drupal7
$ drush dl entity views ctools facetapi
$ drush en search_api search_api_views search_api_solr search_api_facetapi entity views views_ui ctools facetapi

(Optionally) I install devel, admin_menu, and disable overlay/toolbar

$ drush dl devel admin_menu
$ drush en devel admin_menu
$ drush dis overlay toolbar

Add the tomcat/solr server to Search API configuration.
- URL: /admin/config/search/search_api
- click on "+ Add Server"
- server name: Solr 3.5.0
- Service class: Solr service
- Solr host: localhost
- Solr port: 8080
- Solr path: /solr
- click Create Server

You should receive some confirmation messages:
The server was successfully created.
The Solr server could be reached (latency: # ms).
If not, ensure tomcat/solr is reachable at the url you specified and the tomcat service is running.

At this point Solr is ready to send/receive data and index content, but there is nothing to index. For this tutorial, I decided to build off of user profiles and store latitude and longitude using the geolocation field module.

$ drush dl geolocation
$ drush en geolocation

Adding some user profile fields:
- URL: /admin/config/people/accounts/fields
- First Name | field_name_first | Text
- Last Name | field_name_last | Text
- Geolocation | field_geolocation | Geolocation | Latitude/Longitude

I then added a bunch of users with latitude/longitude coordinates.
- URL: /admin/people/create
- note: I used Google Geocoding API to fetch the coordinates: http://code.google.com/apis/maps/documentation/geocoding/

Adding the search api index.
- URL: /admin/config/search/search_api
- click "+ Add index"
- Index name: People
- Item type: User
- Server: Solr 3.5.0
- click: Create Index

On the next admin page, you can select which fields to index. For this tutorial, I chose: User ID, Name, Email, URL, First Name, and Last Name. Unfortunately, at the time of writing this, the geolocation lat/lng fields are not exposed to the Entity API. I assume this is a temporary problem, and there are numerous patches in the geolocation issue queue.
@see (for example):
Property Info callback for Entity API - http://drupal.org/node/1366642
Fix for Search API not picking up the entity to index it's fields - http://drupal.org/node/1320564

I copied code directly from the issues queue, made some modifications, and created a custom module to expose the geolocation field data to the entity api module. In addition, I added a new property "lat_lon" that concatenates lat and lng together with a comma. @see: http://wiki.apache.org/solr/SpatialSearch

<?php
/**
 * Implements hook_field_info_alter()
 */
function MYMODULE_field_info_alter(&$info) {
  if (isset($info['geolocation_latlng'])) {
    $info['geolocation_latlng']['property_type'] = 'geolocation';
    $info['geolocation_latlng']['property_callbacks'] = array('geolocation_property_info_callback');
  }
}

function geolocation_property_info_callback(&$info, $entity_type, $field, $instance, $field_type) {
  $name = $field['field_name'];
  $property = &$info[$entity_type]['bundles'][$instance['bundle']]['properties'][$name];

  $property['type'] = ($field['cardinality'] != 1) ? 'list<geolocation>' : 'geolocation';
  $property['getter callback'] = 'entity_metadata_field_verbatim_get';
  $property['setter callback'] = 'entity_metadata_field_verbatim_set';
  $property['auto creation'] = 'geolocation_default_values';
  $property['property info'] = geolocation_data_property_info();

  unset($property['query callback']);
}

function geolocation_default_values() {

  return array(
    'lat' => '',
    'lng' => '',
    'lat_sin' => '',
    'last_name' => '',
    'lat_cos' => '',
    'lat_rad' => '',
    'lat_lon' => '',
  );

}

function geolocation_data_property_info($name = NULL) {

  // Build an array of basic property information for the geolocation field.
  $properties = array(
    'lat' => array(
      'label' => t('Latitude'),
    ),
    'lng' => array(
      'label' => t('Longitude'),
    ),
    'lat_sin' => array(
      'label' => t('Sine of Latitude'),
    ),
    'lat_cos' => array(
      'label' => t('Cosine of Latitude'),
    ),
    'lat_rad' => array(
      'label' => t('Radian Latitude'),
    ),
    'lat_lon' => array(
      'label' => t('Latitude,Longitude'),
    ),
  );

  // Add the default values for each of the address field properties.
  foreach ($properties as $key => &$value) {
    
    switch ($key) {
    
      case 'lat_lon':
        $value += array(
          'description' => !empty($name) ? t('!label of field %name', array('!label' => $value['label'], '%name' => $name)) : '',
          'type' => 'text',
          'getter callback' => '_MYMODULE_geolocation_entity_property_verbatim_get',
          'setter callback' => '_MYMODULE_geolocation_entity_property_verbatim_set',
        );
        break;
    
      default:
        $value += array(
          'description' => !empty($name) ? t('!label of field %name', array('!label' => $value['label'], '%name' => $name)) : '',
          'type' => 'text',
          'getter callback' => 'entity_property_verbatim_get',
          'setter callback' => 'entity_property_verbatim_set',
        );
        break;
    
    }

 }

 return $properties;
}


function _MYMODULE_geolocation_entity_property_verbatim_get($data, array $options, $name, $type, $info) {
  if (is_array($data) && isset($data['lat']) && isset($data['lng'])) {
    return $data['lat'] . ',' . $data['lng'];
  }
  return '';
}

function _MYMODULE_geolocation_entity_property_verbatim_set(&$data, $name, $value, $langcode, $type, $info) {
  // TODO
  return;
}
?>

I added this code to a custom module, renamed function calls (as necessary), and enabled. Update the solr index to add the new fields to the index.
- URL: /admin/config/search/search_api/index/people/fields
- Expand "Add Related Fields"
- Choose Geolocation, click Add fields
The above will expose the following fields now available to the index:
- Geolocation » Latitude
- Geolocation » Longitude
- Geolocation » Sine of Latitude
- Geolocation » Cosine of Latitude
- Geolocation » Radian Latitude
- Geolocation » Latitude,Longitude
Enable "Geolocation » Latitude,Longitude" and save changes.

Index the content.
- URL: /admin/config/search/search_api/index/people/status
- Click: Index now
- note: if you had already indexed the content, you'll probably need to clear it first
In my environment, I got the following confirmation message:
Successfully indexed 7 items.

I find it to be very helpful to verify the xml response from Solr directly after making changes to the index/schema.
The following URL structure will query solr for all results and return all fields:

Ex URL: http://drupal7.vm:8080/solr/select/?q=&fl=*

A sample XML document response.

<doc>
  <str name="f_ss_search_api_language"/>
  <str name="f_ss_url">http://drupal7.vm/user/3</str>
  <str name="id">people-3</str>
  <str name="index_id">people</str>
  <long name="is_uid">3</long>
  <str name="item_id">3</str>
  <arr name="spell">
    <str>nashua</str>
    <str>nashua@example.com</str>
    <str>nashua</str>
    <str>nashua</str>
    <str>42.933692,-72.278141</str>
  </arr>
  <str name="ss_search_api_id">3</str>
  <str name="ss_search_api_language"/>
  <str name="ss_url">http://drupal7.vm/user/3</str>
  <arr name="t_field_geolocation:lat_lon">
    <str>42.933692,-72.278141</str>
  </arr>
  <arr name="t_field_name_first">
    <str>nashua</str>
  </arr>
  <arr name="t_field_name_last">
    <str>nashua</str>
  </arr>
  <arr name="t_mail">
    <str>nashua@example.com</str>
  </arr>
  <arr name="t_name">
    <str>nashua</str>
  </arr>
</doc>

Take note the field name in the following XML, it is used in the next file edit.

<arr name="t_field_geolocation:lat_lon">
  <str>42.933692,-72.278141</str>
</arr>

Update the solr schema.xml configuration and add the geospatial fieldType and field data.

# Edit file: /var/lib/tomcat6/solr/conf/schema.xml
# Just prior to the closing "</types>" tag, I inserted: (around line 287)
    <fieldType name="point" class="solr.PointType" dimension="2" subFieldSuffix="_d"/>
    <fieldType name="location" class="solr.LatLonType" subFieldSuffix="_coordinate"/>
    <fieldtype name="geohash" class="solr.GeoHashField"/>

# And, just after the opening "<fields>" tag, I inserted: 
    <field name="t_field_geolocation:lat_lon" type="location" indexed="true" stored="true"/>
    <dynamicField name="*_coordinate"  type="tdouble" indexed="true"  stored="false"/>

Restart Tomcat

$ /etc/init.d/tomcat6 restart

Since the schema and solr data types have been updated, the content will have to be re-indexed.
- URL: /admin/config/search/search_api/index/people/status
- click: Clear index
- click: Index now

Returning to the solr query above will now show updated xml: (note: no longer an array)

<str name="t_field_geolocation:lat_lon">42.933692,-72.278141</str>

Verify the native solr geospatial searching is working using the following query syntax:
URL: http://drupal7.vm:8080/solr/select/?q=&fl=*&fq={!geofilt sfield=t_field_geolocation:lat_lon pt=42.933692,-72.278141 d=100}
By putting a distance parameter of 100 (kilometers) and Nashua NH coordinates, I get 2 results: Nashua and Portsmouth, awesome.

Create a solr integrated view.
- URL: /admin/structure/views/add
- View name: People
- Show: People
- Create a Page [checked]
- Path: people
- Continue & edit
Note: at this point, you have full reign over view configuration. For this tutorial, I set the format to Grid, and added some fields:
- Geolocation: Latitude,Longitude (indexed)
- Indexed User: Email
- Indexed User: First Name
- Indexed User: Last Name
- Indexed User: Name
Save the view when edits are complete.

Browsing to the view will show something like this:
Ex URL: http://drupal7.vm.people


The next chunk of custom code modifies the solr query executed and adds geospatial filtering.
@see: hook_search_api_solr_query_alter(array &$call_args, SearchApiQueryInterface $query)

<?php
function MYMODULE_search_api_solr_query_alter(array &$call_args, SearchApiQueryInterface $query) {

  $lat = 42.933692;
  $lng = -72.278141;
  $distance = 100;

  $call_args['params']['fq'][] = "{!geofilt sfield=t_field_geolocation:lat_lon pt={$lat},{$lng} d={$distance}}";

}
?>

The above code will limit the view's results using the hardcoded coordinates.


Clearly, it works but there are loose ends to tie..
- automatically fetch a user's coordinates to store in the geolocation field
- add a search form to the people view page to allow the user to search for a location (instead of hard coded coordinates, blah)
- translate the user's location search input to coordinates using an API

Hopefully, I can find more time to elaborate on this tutorial in the near future! Cheers.

In this tutorial, I'll show how you can make a SOAP call from a Drupal page callback using the nusoap library.

For this example, I decided to create a sample soap server instance for testing purposes. I created the following directory structure for my new module: sites/all/modules/custom/nusoap/. I then download the nusoap library (nusoap-0.7.3.zip), extracted the archive, and put the "lib" folder in my module directory (sites/all/modules/custom/nusoap/lib).

In this directory, I created a file called "soap-server.php" to contain my soap server instance and added the following code:

<?php
// define namespace
define('NUSOAP_NAME_SPACE', 'erl.dev');

// define path to nusoap library file
$nu_soap_path = 'lib/nusoap.php';

// ensure nu_soap library exsists
if (!file_exists($nu_soap_path)) {
  die('An error has occurred initializing the soap server.');
}

// include nu_soap library
require_once ($nu_soap_path);

// create new soap server instance
$soap_server = new nusoap_server();

// configure wsdl
$soap_server->configureWSDL(NUSOAP_NAME_SPACE, 'urn:'. NUSOAP_NAME_SPACE);

// add a custom data type: person
$soap_server->wsdl->addComplexType(
  'person',
  'complexType',
  'struct',
  'all',
  '',
  array(
    'firstName' => array(
      'name' => 'firstName',
      'type' => 'xsd:string',
    ),
    'lastName' => array(
      'name' => 'lastName',
      'type' => 'xsd:string',
    ),
  ) 
);

// register method: personTransfer
$soap_server->register(
  // method name
  'personTransfer',
  // input args
  array('person' => 'tns:person'),
  // output args
  array('return' => 'tns:person'),
  // namespace
  'uri:'. NUSOAP_NAME_SPACE,
  // SOAPAction
  'uri:'. NUSOAP_NAME_SPACE .'#personTransfer',
  // style
  'rpc',
  // use
  'encoded'
);

// process raw post data
$HTTP_RAW_POST_DATA = isset($HTTP_RAW_POST_DATA) ? $HTTP_RAW_POST_DATA : '';
$soap_server->service($HTTP_RAW_POST_DATA);

/**
 * Define soap methods
 */
function personTransfer($person = array()) {

  // per testing, modify data
  foreach ($person as $key => $value) {
    $person[$key] = $value . "!";
  };

  return $person;

}
?>

Now, if I browse directly to my soap-server.php file (for example: http://drupal.erl.dev/sites/all/modules/custom/nusoap/soap-server.php), I see the following screen:



Clicking on the WSDL link will show my automatically generated WSDL/XML; clicking on the method name "personTransfer" will show more details about the soap server method. Thanks nusoap!

Next, I created the custom Drupal module file...

<?php
// define namespace
define('NUSOAP_NAME_SPACE', 'erl.dev');

/**
 * Implements hook_perm()
 */
function nusoap_perm() {
  return array('access soap');
}

/**
 * Implements hook_menu()
 */
function nusoap_menu() {

  $items = array();
  
  $items['soap-client'] = array(
    'title' => t('Soap client'),
    'description' => t('Soap client'),
    'page callback' => 'nusoap_page_callback_soap_client',
    'access arguments' => array('access soap'),
    'type' => MENU_CALLBACK
  );
  
  return $items;

}

/**
 * Implements custom page callback for soap client
 */
function nusoap_page_callback_soap_client() {

  // include nu_soap library
  require_once(drupal_get_path('module', 'nusoap') .'/lib/nusoap.php');

  // define wsdl path
  $wsdl_path = 'http://' . $_SERVER['HTTP_HOST'] . base_path() . drupal_get_path('module', 'nusoap') . '/soap-server.php?wsdl';

  // create new soap client instance
  $soap_client = new nusoap_client($wsdl_path, true);

  // check for error
  $error = $soap_client->getError();
  if ($error) {
    // handle error
  }

  // define method arguments
  $args = array(
    'person' => array(
      'firstName' => 'Eric',
      'lastName' => 'London'
    )
  );

  // call soap server method
  $result = $soap_client->call('personTransfer', $args);

  // debug output:
  $output = "";
  $output .= "<pre>";
  $output .= "SENT: ";
  $output .= print_r($args, true);
  $output .= "RECEIVED: ";
  $output .= print_r($result, true);
  $output .= "</pre>";
    
  return $output;
 
}
?>

Browsing to the new page callback (ex: http://drupal.erl.dev/soap-client) shows the following debug output. sweet.