Blogsplat

Recently I’ve had to use delayed_job (which you should be using too!), and rather than have capistrano start delayed job on deploy, I figured I should look into using a process manager, and since I’m a Ruby guy, I decided to see what was available to “gem install”.

The general consensus is that bluepill is the way to go for simple process management with a Ruby DSL. Basically, you tell bluepill what to run, how to know if it’s running, how to stop it, and when to kill it, and it will take care of doing all that for you.

The syntax is clean and straightforward. Here is my config/delayed_job.pill file:

APP_ROOT='/var/www/apps/application.com'
RAILS_ROOT='/var/www/apps/application.com/current'
RAILS_ENV='production'

Bluepill.application("application.com'") do |app|
  app.process("delayed_job") do |process|
    process.working_dir = RAILS_ROOT

    process.start_command = "/usr/bin/env RAILS_ENV=#{RAILS_ENV} #{RAILS_ROOT}/script/delayed_job start"
    process.stop_command  = "/usr/bin/env RAILS_ENV=#{RAILS_ENV} #{RAILS_ROOT}/script/delayed_job stop"
    process.pid_file = "#{APP_ROOT}/shared/pids/delayed_job.pid"
    process.start_grace_time = 15.seconds
    process.stop_grace_time = 15.seconds
    process.uid = "deploy"
    process.gid = "deploy"

    process.checks :cpu_usage, :every => 10.seconds, :below => 5, :times => 3
    process.checks :mem_usage, :every => 10.seconds, :below => 100.megabytes, :times => [3,5]
  end
end

This configuration will keep my delayed job running smoothly, and kill it if it gets out of hand. Notice the grace times. Since this is a rails process, I let bluepill know that it might take a bit of time to complete each of these commands.

To load this file into bluepill, just run bluepill load config/delayed_job.pill from your rails directory.

Now, I could leave it at that and just have a capistrano hook that makes sure it runs on deploy, but instead I decided to make the system (linux) do that for me. Ubuntu uses a more modern approach than an old-school unix by using a software called upstart to manage processes.

Upstart has a concise language for configuring starting/stopping, and restarting processes. In a way, it’s like bluepill, but config files live in /etc/init.

I believe I might be able to do all of this from upstart, but I didn’t want to put a bunch of app specific init files in my /etc/init, though I may investigate this in the future.

Anyway, in order to get the system to start up bluepill, and keep it running, I just made a new init file, /etc/init/bluepill.conf with this contents:

# bluepill - process monitor
#
# simple process monitoring tool

description	"simple process monitoring tool"

start on runlevel [2345]
stop on runlevel [!2345]

expect daemon
respawn

env PATH=/opt/ruby-enterprise-1.8.7-2010.01/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

exec /opt/ruby-enterprise-1.8.7-2010.01/bin/bluepill load /var/www/apps/application.com/current/config/delayed_job.pill

And that’s it, you can now run start bluepill which will kick off the bluepill process from upstart, and then bluepill will kick off your delayed_job.

I’m using ruby enterprise, so I needed to include my path in the upstart file (even though I also specified the full path to the bluepill script), but this wouldn’t be necessary if you were running a ruby from /usr or /usr/local.

Try it out, now you can kill either your delayed_job or your bluepill process and it will bounce right back. Cool!