Using Jenkins to build and test a Docker Compose cluster (via Rails, Elasticsearch, and RSpec)

In this post I’ll share a working example that uses Jenkins to build and test your Docker Compose cluster. For this example I created a simple Rails project that connects to Postgresql and Elasticsearch. I used RSpec to test the connectivity of the container services, and then configured Jenkins to build the project and run the test suite.

I installed my Docker dependencies via Brew on OSX.

$ brew list --versions | grep -i docker
docker 17.03.0
docker-compose 1.11.2
docker-machine 0.10.0
docker-machine-nfs 0.4.1

Rails Project

# rvm files
echo docker-jenkins > .ruby-gemset
echo ruby-2.4.0 > .ruby-version
cd .

gem install rails

rails -v
Rails 5.0.2

# new project with postgresql connection
rails new . --api -d postgresq

# init database
rake db:create && rake db:migrate

Add gems, edit file: Gemfile

group :development, :test do
  gem 'rspec-rails'
end

gem 'elasticsearch-model'
gem 'elasticsearch-rails'

Execute bundle install to install the new gems.

Executed rails g rspec:install to generate the default RSpec configuration.

Enabled Elasticsearch logging integration. Edit file: config/application.rb, added:

require 'elasticsearch/rails/instrumentation'

Updated database configuration to use environment variables for connectivity. edit file: config/database.yml

default: &default
  adapter: postgresql
  encoding: unicode
  pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
  host: <%= ENV.fetch('POSTGRESHOST', 'localhost') %>
  password: <%= ENV.fetch('POSTGRESPASS', 'postgres') %>
  username: <%= ENV.fetch('POSTGRESUSER', 'postgres') %>

Add a migration to create a table for a Person model (first and last name), new file: db/migrate/20170315211521_create_people.rb

class CreatePeople < ActiveRecord::Migration[5.0]
  def change
    create_table :people do |t|
      t.string :first_name
      t.string :last_name

      t.timestamps
    end
  end
end

Created the Person model with Elasticsearch integration, new file: app/models/person.rb

class Person < ApplicationRecord
  include Elasticsearch::Model
  include Elasticsearch::Model::Callbacks
end

Added an RSpec model test file for the Person model. I purposely did not mock the backend to ensure the services are connecting properly. new file: spec/models/person_spec.rb

require 'rails_helper'

RSpec.describe Person, type: :model do
  describe 'elasticsearch' do
    before do
      Person.destroy_all
    end

    after do
      person.destroy
    end

    let(:first_name) { "first #{DateTime.now.to_i}" }
    let(:last_name) { "last #{DateTime.now.to_i}" }
    let!(:person) { Person.create!(first_name: first_name, last_name: last_name) }
    let(:get_params) do
      {
        index: Person.index_name, id: person.id
      }
    end
    let!(:document) { Person.__elasticsearch__.client.get(get_params) }

    it 'indexes model data' do
      expect(document['_source']['id']).to eq(person.id)
      expect(document['_source']['first_name']).to eq(first_name)
      expect(document['_source']['last_name']).to eq(last_name)
    end
  end
end

Docker Integration

Add a simple Dockerfile for the Rails container that adds netcat to test for service connectivity, new file: Dockerfile

FROM ruby:2.4.0

RUN apt-get update -qq && apt-get install -y netcat

ENV APP_HOME /rails

WORKDIR $APP_HOME

Created the Docker compose file to define the containers for Rails, Elasticsearch, and Postgresql, new file: docker-compose.yml

version: '2'
services:
  elasticsearch:
    image: elasticsearch:2.4.4
    ports:
      - '9200:9200'
      - '9300:9300'
    volumes:
      - elasticsearch:/usr/share/elasticsearch/data
  postgres:
    environment:
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_USER=postgres
    image: postgres:9.6.2
    ports:
      - '5432:5432'
    volumes:
      - postgres:/var/lib/postgresql/data
  rails:
    build:
      context: .
      dockerfile: Dockerfile
    command: bin/docker-start-rails
    depends_on:
      - elasticsearch
      - postgres
    environment:
      - ELASTICSEARCH_URL=http://elasticsearch:9200
      - POSTGRESHOST=postgres
      - POSTGRESPASS=postgres
      - POSTGRESUSER=postgres
    links:
      - elasticsearch
      - postgres
    ports:
      - '3000:3000'
    volumes:
      - .:/rails
      - bundle:/usr/local/bundle
volumes:
  bundle: {}
  elasticsearch: {}
  postgres: {}

Added a shell script to start rails, new file: bin/docker-start-rails

#!/bin/sh

rm tmp/pids/server.pid

bundle check || bundle install

# wait for postgresql
until nc -vz postgres 5432 2>/dev/null; do
  echo "Postgresql is not ready, sleeping."
  sleep 1
done

# wait for elasticsearch
until nc -vz elasticsearch 9200 2>/dev/null; do
  echo "Elasticsearch is not ready, sleeping."
  sleep 1
done

rake db:create
rake db:migrate

rails s -b 0.0.0.0

I added a simple Ping controller which will be used by Jenkins to ensure the Rails container is up before starting the RSpec test suite. new file: app/controllers/api/pings_controller.rb

class Api::PingsController < ApplicationController
  def show
    render json: 'pong'
  end
end

And the corresponding route for the ping request, edit file: config/routes.rb, added:

Rails.application.routes.draw do
  namespace :api do
    resource :ping, only: :show
  end
end

At this point the application can be functionally tested via: docker-compose build && docker-compose up

Jenkins Integration

I added a bin script as an entry point for Jenkins. The script defines a docker machine called “jenkins”, ensures it is created, running, and active. It stops and removes any existing containers. It then builds the Docker compose cluster, waits for Rails to respond to the ping request, and executes the RSpec test suite. new file: bin/jenkins

#!/bin/sh

set -x

DOCKER_MACHINE=jenkins

EXISTS=$(docker-machine ls --filter name=$DOCKER_MACHINE -q)
if [ "$EXISTS" = "" ]; then
  docker-machine create -d virtualbox $DOCKER_MACHINE
  eval $(docker-machine env $DOCKER_MACHINE)
fi

STATE=$(docker-machine ls --filter name=$DOCKER_MACHINE | awk '{print $4}' | tail -1)
if [ "$STATE" = "Stopped" ]; then
  docker-machine start $DOCKER_MACHINE
  eval $(docker-machine env $DOCKER_MACHINE)
fi

ACTIVE=$(docker-machine ls --filter name=$DOCKER_MACHINE | awk '{print $2}' | tail -1)
if [ "$ACTIVE" = "-" ]; then
  eval $(docker-machine env $DOCKER_MACHINE)
fi

docker stop $(docker ps -a -q)
docker rm $(docker ps -a -q)

docker-compose build
docker-compose up -d

IP=$(docker-machine ip $DOCKER_MACHINE)

# wait for rails
while true
do
	response=$(curl http://$IP:3000/api/ping 2>/dev/null)
  if [ "$response" = "pong" ]; then
    break
  fi
  sleep 1
done

RAILS_CONTAINER=$(docker ps -a | grep -i 3000 | awk '{print $NF}')

docker exec -i $RAILS_CONTAINER rspec
RSPEC_RESULT=$?

docker stop $(docker ps -a -q)
docker rm $(docker ps -a -q)

exit $RSPEC_RESULT

Source code on Github

I installed Jenkins locally via brew:

brew install jenkins

brew list --versions | grep -i jenkins
jenkins 2.50

The following images shows the settings I used for the Jenkins Docker Compose project. I integrated with my Github repo and set Jenkins to poll every 15 minutes for changes. I added a build step to timeout after 10 minutes, and execute the following:

#!/bin/bash -l
bin/jenkins

Jenkins Docker Rails

Eric London