Rails 4 Elasticsearch integration with dynamic facets and filters via model concern

In this blog post, I’ll share some code that integrates Rails 4 models with Elasticsearch to provide a search form with dynamically generated facets and filters. The below code shows how you can use class introspection with Rails models to create a searchable framework without hardcoding your model attributes.

Install JRuby via RVM. note: JRuby is optional, this code works with MRI Ruby as well.

rvm get stable
rvm install jruby

Install Elasticsearch via homebrew.

# install
brew install elasticsearch

# start
elasticsearch --config=/usr/local/opt/elasticsearch/config/elasticsearch.yml

# note: browse to http://localhost:9200 to ensure running

Setup new Rails project.

mkdir rails_elasticsearch
echo "jruby-1.7.13" > rails_elasticsearch/.ruby-version
echo "rails_elasticsearch" > rails_elasticsearch/.ruby-gemset
cd rails_elasticsearch
gem install rails
rails new .
rake db:migrate

Added gems, edited file: Gemfile. Executed: bundle install

# elasticsearch
gem 'elasticsearch-model', git: 'git://github.com/elasticsearch/elasticsearch-rails.git'
gem 'elasticsearch-rails', git: 'git://github.com/elasticsearch/elasticsearch-rails.git'

# for seeds
gem 'random-word', group: [:development, :test]

# development/debug gems
group :development do
  gem 'pry'
  gem 'pry-rails'
  gem 'quiet_assets'
  gem 'better_errors', '< 2' # https://github.com/charliesome/better_errors/commit/a449f136124f2933a39f038249693bda381cd097
end

# bootstrap gems
gem 'bootstrap-sass', '~> 3.2.0'
gem 'autoprefixer-rails'

Added example models:

rails g scaffold Person first_name:string last_name:string age:integer city:string state:string

rails g scaffold Thing name:string description:string person_id:integer

rake db:migrate

Created rails model concern to integrate models with Elasticsearch. The Rails concern code provided is based on this example template provided by the Elasticsearch gem. I revised it and removed the hardcoded model attributes.

Create new file: app/models/concerns/elasticsearch_searchable.rb

require 'active_support/concern'

module ElasticsearchSearchable
  extend ActiveSupport::Concern

  # Search model instance code:
  included do

    # require and include Elasticsearch libraries
    require 'elasticsearch/model'
    include Elasticsearch::Model
    #include Elasticsearch::Model::Callbacks
    include Elasticsearch::Model::Indexing

    # index document on model touch
    # @see: http://api.rubyonrails.org/classes/ActiveRecord/Associations/ClassMethods.html
    after_touch() { __elasticsearch__.index_document }

    # Customize the JSON serialization for Elasticsearch
    def as_indexed_json(options={})

      # define JSON structure (including nested model associations)
      _include = self.class.reflect_on_all_associations.each_with_object({}) {|a,hsh|
        hsh[a.name] = {}
        hsh[a.name][:only] = a.klass.attribute_names
      }

      self.as_json(include: _include)
    end

  end

  # Search model class methods
  module ClassMethods

    # todo/note: params processing is a controller function.
    def search_params(params={})
      return [nil,nil] if params.blank? || params[:search].blank?
      p = params[:search].dup
      q = p.delete(:q)
      [q, p]
    end

    # define search method to be used in Rails controller
    def search(query=nil, options={})

      options ||= {}

      # setup empty search definition
      @search_definition = {
        query: {},
        filter: {},
        facets: {},
      }

      # Prefill and set the filters (top-level `filter` and `facet_filter` elements)
      __set_filters = lambda do |key, f|

        @search_definition[:filter][:and] ||= []
        @search_definition[:filter][:and]  |= [f]

        @search_definition[:facets][key.to_sym][:facet_filter][:and] ||= []
        @search_definition[:facets][key.to_sym][:facet_filter][:and]  |= [f]
      end

      # facets
      @search_definition[:facets] = search_facet_fields.each_with_object({}) do |a,hsh|
        hsh[a.to_sym] = {
          terms: {
            field: a
          },
          facet_filter: {}
        }
      end

      # query
      unless query.blank?
        @search_definition[:query] = {
          bool: {
            should: [
              { multi_match: {
                  query: query,
                  # limit which fields to search, or boost here:
                  fields: search_text_fields,
                  operator: 'and'
                }
              }
            ]
          }
        }
      else
        @search_definition[:query] = { match_all: {} }
      end

      # add filters for facets
      options.each do |key,value|
        next unless search_facet_fields.include?(key)

        f = { term: { key.to_sym => value } }

        __set_filters.(key, f)

      end

      # execute Elasticsearch search
      __elasticsearch__.search(@search_definition)

    end

    private

    # return array of model attributes to search on
    def search_text_fields
      self.content_columns.select {|c| [:string,:text].include?(c.type) }.map {|c| c.name }
    end

    # return array of model attributes to facet
    def search_facet_fields
      self.content_columns.select {|c| [:boolean,:decimal,:float,:integer,:string,:text].include?(c.type) }.map {|c| c.name }
    end

  end

end

Added model associations, and searchable concern integration.

Edited file: app/models/thing.rb

class Thing < ActiveRecord::Base

  belongs_to :person, touch: true

  include ElasticsearchSearchable

end

Edited file: app/models/person.rb

class Person < ActiveRecord::Base

  has_many :things

  after_update { self.things.each(&:touch) }

  include ElasticsearchSearchable

end

Seeded data. Edited file: db/seeds.rb

CityState = Struct.new(:city, :state)

city_states = [
  CityState.new('Boston',      'Massachusetts'),
  CityState.new('Worcester',   'Massachusetts'),
  CityState.new('Providence',  'Rhode Island'),
  CityState.new('Springfield', 'Massachusetts'),
  CityState.new('Bridgeport',  'Connecticut'),
  CityState.new('New Haven',   'Connecticut'),
  CityState.new('Hartford',    'Connecticut'),
  CityState.new('Stamford',    'Connecticut'),
  CityState.new('Waterbury',   'Connecticut'),
  CityState.new('Manchester',  'New Hampshire'),
  CityState.new('Lowell',      'Massachusetts'),
  CityState.new('Cambridge',   'Massachusetts'),
  CityState.new('New Bedford', 'Massachusetts'),
  CityState.new('Brockton',    'Massachusetts'),
  CityState.new('Quincy',      'Massachusetts'),
  CityState.new('Lynn',        'Massachusetts'),
  CityState.new('Fall River',  'Massachusetts'),
  CityState.new('Nashua',      'New Hampshire'),
  CityState.new('Norwalk',     'Connecticut'),
  CityState.new('Newton',      'Massachusetts'),
]

(1..10).each do

  city_state = city_states.sample

  # create person
  person = Person.create({
    first_name: RandomWord.nouns.next,
    last_name:  RandomWord.nouns.next,
    age:        Random.rand(1..100),
    city:       city_state.city,
    state:      city_state.state,
  })

  # create things
  things = (1..10).map do
    Thing.create({
      name:        RandomWord.nouns.next,
      description: RandomWord.nouns.next,
    })
  end

  person.things = things
  person.save

end

Executed rake db:seed to populate development database.

Curl Elasticsearch to review JSON structure after seeding data. Note: the JSON response “things” array truncated for the sake of brevity. Executed: curl http://localhost:9200/people/_search?size=1 | python -mjson.tool

{
    "_shards": {
        "failed": 0,
        "successful": 5,
        "total": 5
    },
    "hits": {
        "hits": [
            {
                "_id": "4",
                "_index": "people",
                "_score": 1.0,
                "_source": {
                    "age": 41,
                    "city": "Brockton",
                    "created_at": "2014-08-28T19:29:15.264Z",
                    "first_name": "monetization",
                    "id": 4,
                    "last_name": "tympanuchus_cupido",
                    "state": "Massachusetts",
                    "things": [
                        {
                            "created_at": "2014-08-28T19:29:15.288Z",
                            "description": "rate_of_return",
                            "id": 31,
                            "name": "alpine_woodsia",
                            "person_id": 4,
                            "updated_at": "2014-08-28T19:29:15.632Z"
                        }
                    ],
                    "updated_at": "2014-08-28T19:29:15.814Z"
                },
                "_type": "person"
            }
        ],
        "max_score": 1.0,
        "total": 10
    },
    "timed_out": false,
    "took": 1
}

Created a search controller with an index method. Executed: rails g controller Search index

Revised search controller index method, file: app/controllers/search_controller.rb

class SearchController < ApplicationController

  def index
    @search = Person.search( *Person.search_params(params) )
  end

end

Revised search index view to provide all search form, layout, and facet markup. Edited file: app/views/search/index.html.erb

<h1>Search</h1>

<div class='row'>

  <div class="col-md-3">

    <%= form_tag search_index_path, method: :get do %>

      <div class="form-group">
        <label>Search</label>
        <input type="search" class="form-control" name="search[q]" placeholder="Search" value="<%= (params[:search].present? && params[:search][:q].present?) ? params[:search][:q] : "" %>">
      </div>

      <div class='form-group'>
        <button type="submit" class="btn btn-default" id="search_btn">Search</button>
        <%= link_to 'Reset Search', search_index_path, {class: 'btn btn-default', role: :button} %>
      </div>

      <%# facets %>

      <% @search.response.facets.each do |facet,facet_data| %>
        <div class='form-group'>
          <label><%= ActiveSupport::Inflector.humanize(facet) %></label>
          <select class='form-control facet_select' size='5' name="search[<%= facet %>][]" multiple>
            <% facet_data.terms.each do |f| %>
              <% selected = (params[:search].present? && params[:search][facet].present? && params[:search][facet].include?(f[:term].to_s)) ? true : false %>
              <option data-selected="<%= selected %>" value="<%= f[:term] %>" <%= selected ? "selected" : "" %> >
                <%= f[:term] %>
                (<%= f[:count] %>)
              </option>
            <% end %>
          </select>
        </div>
      <% end %>

    <% end %>

  </div>

  <div class="col-md-9">

    <%# results %>

    <table class="table table-striped table-hover">
      <thead>
        <tr>
          <% @search.klass.attribute_names.each do |field| %>
            <th>
              <%= field %>
            </th>
          <% end %>
        </tr>
      </thead>
      <tbody>
        <% @search.results.each do |result| %>
          <tr>
            <% @search.klass.attribute_names.each do |field| %>
              <td>
                <%= result._source[field] if result._source.respond_to?(field) %>
              </td>
            <% end %>
          </tr>
        <% end %>
      </tbody>
    </table>

  </div>

</div>

Added 2 javascript click events for facets and the search button, edited file: app/assets/javascripts/search.js

$(document).ready(function() {

  // click event of search phrase
  $('#search_btn').click(function(){
    $form = $(this).parents('form');
    $form.find('.facet_select option').prop('selected', false);
    $form.submit();
    return false;
  });

  // click event of facet select option
  $('.facet_select option').click(function(){
    if ($(this).data('selected')) {
      $(this).prop('selected', false);
    }
    $(this).parents('form').submit();
  });

});

As seen in the above code, the ElasticsearchSearchable concern, search controller, and views do not contain hard-coded model attributes. Clearly this code will require customizations based on the project, but it’s a proof of concept. Here is a screenshot in action, url: http://localhost:3000/search/index

The Bootstrap integration, layout, modifications to CSS/markup/views/etc, and everything else can be seen in the source code at Github.