Dockerize a Rails development environment integrated with Postgresql, Redis, and Elasticsearch using Docker Compose

In this post I’ll share an example configuration to Docker-ize a Rails development environment integrated with Postgresql, Redis, and Elasticsearch using Docker Compose. In this guide I’ll focus on interoperability between running your code in Docker and locally.

I started development using the pre-built OSX download for Docker which uses Xhyve. I had some problems with container volume peristence, so for this post I’ll demonstrate how to use VirtualBox as your docker-machine.

Initial Docker installation

# install docker packages via brew
brew install docker docker-machine docker-compose

# create a new virtualbox docker-machine
docker-machine create -d virtualbox docker-machine

# start new docker-machine (if not already running)
docker-machine start docker-machine

# set docker environment variables.
# You might want to put this in your shell profile, ex: ~/.profile
eval $(docker-machine env docker-machine)

Initial project creation

# create directory for Rails and Docker files
mkdir rails_dev

# create RVM files. Note: these will not be used by Docker, just for local development
echo ruby-2.3.1 > rails_dev/.ruby-version
echo rails_docker > rails_dev/.ruby-gemset
cd rails_dev

# install rails gem (latest was 5.0.0.1) and create the new project files
gem install rails
rails new . -d postgresql

Add required gems: devise for authentication, elasticsearch for searching, and redis-rails for sessions/caching. edit file: Gemfile, add:

gem 'devise'
gem 'elasticsearch-model'
gem 'elasticsearch-rails'
gem 'redis-rails'

Normally you could run bundle install to install the new gems, but we can wait for the Docker build to do so.

Rails configuration

Update database configuration to use environment variables. edit file: config/database.yml

default: &default
  adapter: postgresql
  encoding: unicode
  pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
  host: <%= ENV.fetch('POSTGRESQL_HOST', 'localhost') %>
  password: <%= ENV.fetch('POSTGRESQL_PASSWORD', '') %>
  username: <%= ENV.fetch('POSTGRESQL_USERNAME', 'postgres') %>

Set session store to use Redis, edit file: config/initializers/session_store.rb

Rails.application.config.session_store :redis_store, servers: "redis://#{ENV.fetch('REDIS_HOST', 'localhost')}:6379/0/session"

Set cache store to use Redis as well, edit file: config/environments/development.rb

# replace:
config.cache_store = :memory_store

# with:
config.cache_store = :redis_store, "redis://#{ENV.fetch('REDIS_HOST', 'localhost')}:6379/0/cache", { expires_in: 90.minutes }

Create a Redis initializer to instantiate a Redis client. new file: config/initializers/redis.rb

module RedisClient
  class << self
    def redis
      @redis ||= Redis.new(host: ENV.fetch('REDIS_HOST', 'localhost'))
    end
  end
end

Run generator for Devise installation and create the User model:

rails generate devise:install
rails generate devise User

Create an initializer for Elasticsearch to set the client host from the environment variable, new file: config/initializers/elasticsearch.rb

Elasticsearch::Model.client = Elasticsearch::Client.new(host: ENV.fetch('ELASTICSEARCH_HOST', 'localhost'))

Add Elasticsearch logging, edit file: config/application.rb, add:

require 'elasticsearch/rails/instrumentation'

Docker integration

At this point the Rails configuration and project files are ready to live in a Docker container.

Create the Dockerfile which will be used to create the Rails container, new file: Dockerfile.

FROM ruby:2.3.1

RUN apt-get update -qq && apt-get install -y build-essential libpq-dev nodejs netcat

RUN mkdir /rails
WORKDIR /rails
ADD Gemfile /rails/Gemfile
ADD Gemfile.lock /rails/Gemfile.lock
RUN bundle install
ADD . /rails

Read more about using Docker Compose with Rails in this article.

And the contents of docker-compose.yml below. Some key points to highlight:

top level volumes are used for persistence
the web container “links” and depends on the postgresql, elasticsearch, and redis containers
each container’s main service is exposing the default ports for connectivity
environment variables are set in the main web container

version: '2'
services:
  elasticsearch:
    image: elasticsearch
    ports:
      - "9200:9200"
      - "9300:9300"
    volumes:
      - elasticsearch:/usr/share/elasticsearch/data
  postgres:
    image: postgres
    ports:
      - "5432:5432"
    volumes:
      - postgres:/var/lib/postgresql/data
  redis:
    image: redis
    ports:
      - "6379:6379"
    volumes:
      - redis:/data
  web:
    build: .
    command: bin/docker-start
    environment:
      - ELASTICSEARCH_HOST=elasticsearch
      - POSTGRESQL_HOST=postgres
      - POSTGRESQL_PASSWORD=
      - POSTGRESQL_USERNAME=postgres
      - REDIS_HOST=redis
      - DEVISE_ADMIN_EMAIL=admin@example.com
    links:
      - postgres
      - elasticsearch
      - redis
    volumes:
      - .:/rails
    ports:
      - "3000:3000"
    depends_on:
      - postgres
      - elasticsearch
      - redis
volumes:
  elasticsearch: {}
  postgres: {}
  redis: {}

Here is the script I chose to execute when the main web container starts, new file: bin/docker-start

#!/bin/sh

set -x

# wait for postgresql
until nc -vz $POSTGRESQL_HOST 5432; do
  echo "Postgresql is not ready, sleeping..."
  sleep 1
done
echo "Postgresql is ready, starting Rails."

# wait for elasticsearch
until nc -vz $ELASTICSEARCH_HOST 9200; do
  echo "Elasticsearch is not ready, sleeping..."
  sleep 1
done
echo "Elasticsearch is ready, starting Rails."

# optional
# rm /rails/tmp/pids/server.pid

# setup database and start puma
RAILS_ENV=development bundle exec rake db:create
RAILS_ENV=development bundle exec rake db:migrate
RAILS_ENV=development bundle exec rake db:seed
RAILS_ENV=development bundle exec rails s -p 3000 -b '0.0.0.0'

Running Docker

Docker can now be built and started via:

docker-compose build
docker-compose up

You can look up the docker machine IP and open in a browser via:

open "http://`docker-machine ip docker-machine`:3000"

Connectivity Test Code

To test that the containers are working together, I added a model and controller to execute some statements and display the results.

New migration to create the table for TestModel, new file: db/migrate/20161029123507_create_test_models.rb

class CreateTestModels < ActiveRecord::Migration[5.0]
  def change
    create_table :test_models do |t|
      t.string :title, null: false
      t.text :body, null: false

      t.timestamps
    end
  end
end

Integrate the TestModel with Elasticsearch and Redis, file: app/models/test_model.rb

require 'elasticsearch/model'

class TestModel < ApplicationRecord
  include Elasticsearch::Model
  include Elasticsearch::Model::Callbacks

  validates :body, presence: true
  validates :title, presence: true

  REDIS_COUNTER_KEY = "#{self}:counter"

  def self.increment_count
    RedisClient.redis.incr REDIS_COUNTER_KEY
  end

  def self.index_mapping
    __elasticsearch__.client.perform_request('GET', "#{index_name}/_mapping", {}, nil).body[index_name]['mappings'][ActiveSupport::Inflector.singularize(index_name)]
  end

  def self.search(query = nil, options = {})
    search_definition = {
      size: options[:size] || 10,
    }

    if query.present?
      search_definition[:query] = {
        bool: {
          should: [
            multi_match: {
              query: query,
              fields: %w(title body),
              operator: 'and'
            }
          ]
        }
      }
    end

    __elasticsearch__.search(search_definition).response.hits
  end
end

Create a Pages controller to execute some test queries and display the results, new file: app/controllers/pages_controller.rb

class PagesController < ApplicationController
  before_action :authenticate_user!

  def index
    # postgresql model data
    @test_models = TestModel.all

    # elasticsearch data
    @index_mapping = TestModel.index_mapping
    @search_results = TestModel.search('test')

    # redis data
    @increment_count = TestModel.increment_count
    @redis_keys = RedisClient.redis.keys
  end
end

The corresponding view ERB file: app/views/pages/index.html.erb

<h2>Test Status</h2>

<pre>
# Redis counter:
<%= @increment_count %>

# Redis keys:
<%= @redis_keys.join("\n") %>

# Model data:
<%= JSON.pretty_generate(@test_models.map(&:attributes)) %>

# Index mapping:
<%= JSON.pretty_generate(@index_mapping) %>

# Search results:
<%= JSON.pretty_generate(@search_results) %>
</pre>

Added routes for the new controller, file: config/routes.rb

Rails.application.routes.draw do
  devise_for :users
  resources :pages, only: [:index]
  root 'pages#index'
end

Last I created a simple seeds script to create a User and TestModel. edit file: db/seeds.rb

User.find_or_create_by!({email: ENV.fetch('DEVISE_ADMIN_EMAIL', 'admin@example.com')}) do |user|
  user.password = 'password'
end

TestModel.find_or_create_by!({title: 'test title', body: 'test body'})

I executed: open "http://`docker-machine ip docker-machine`:3000" to open the Rails docker app. Here is the sample output from the Rails container:

Docker Rails Ouput

To list the created docker containers, execute: docker ps -a

docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS              PORTS                                            NAMES
e088939e2f07        railsdev_web        "bin/docker-start"       About a minute ago   Up About a minute   0.0.0.0:3000->3000/tcp                           railsdev_web_1
74867756a019        elasticsearch       "/docker-entrypoint.s"   About a minute ago   Up About a minute   0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   railsdev_elasticsearch_1
99c4f8208cc1        postgres            "/docker-entrypoint.s"   About a minute ago   Up About a minute   0.0.0.0:5432->5432/tcp                           railsdev_postgres_1
c2627ec9928f        redis               "docker-entrypoint.sh"   About a minute ago   Up About a minute   0.0.0.0:6379->6379/tcp                           railsdev_redis_1

Eric London