So: you’re now hip to doing your heavy lifting in the background, via a queueing system like Delayed Job. If not, read The Rails Way's guide to Delayed Job, watch the Railscast about Delayed Job, or read my post on building a queue-backed feed reader.
I recommend DJ as a ideal way to get started with background jobs, because it introduces no additional moving parts. It does this by using the database for interprocess communication. But databases are not intended to be used as a message bus - this is an example of SQL databases as an overapplied solution.
So if you’re comfortable with the concept of using a queueing system to run background jobs, and are ready to go to the next level of scalability, performance, and robustness, where to from here? It’s time to get your hands dirty with a serious queueing tool: RabbitMQ.
AMQP
RabbitMQ is an Erlang implementation of AMQP, a protocol designed for low-latency, high-reliability asynchronous messaging. The protocol has many facets (queues, exchanges, routing keys, bindings) that can be combined to get different results. For example, broadcast messages (pubsub style) vs. durable queues with guaranteed delivery to a single recipient. For this post, I’ll be focus on the latter category.
Minion
Minion is a Sinatra-style DSL for queueing and consuming jobs over AMQP, written by Heroku co-founder Orion Henry. Queueing a job:
Minion.enqueue('make.sandwich', { 'for' => 'me', 'with' => 'bread' })
Consuming a job from a background worker:
job 'make.sandwich' do |args|
Sandwich.make(args['for'], args['with'])
end
Simple, descriptive, elegant. I’m really digging Minion.
Porting QFeedreader from DJ to Minion
QFeedreader is an example app for reading feeds that I wrote as part of the two part guide to using background jobs and DJ. You can see a running demo here.
Porting it from Delayed Job to Minion was quite easy. I removed the DJ plugin and vendored Minion and its dependency gems (tmm1-amqp, bunny). With that setup done, the changes in my app code were minor. For example, this:
Delayed::Job.enqueue self
…became:
Minion.enqueue('feed.fetch', :url => url)
See the full diff, and the finished app.
Running QFeedreader on Minion/RabbitMQ
Here’s how you can boot QFeedreader on your local workstation.
1) Install Erlang
If you don’t already have Erlang, install it using MacPorts:
$ sudo port install erlang
Or on Ubuntu:
$ apt-get install erlang
2) Run RabbitMQ
Download, build, and run an instance of the RabbitMQ daemon:
$ wget http://www.rabbitmq.com/releases/rabbitmq-server/v1.6.0/rabbitmq-server-1.6.0.tar.gz
Saving to: `rabbitmq-server-1.6.0.tar.gz'
100%[==========================================================>] 130,616 24.2K/s in 4.7s
2009-09-03 15:56:02 (26.9 KB/s) - `rabbitmq-server-1.6.0.tar.gz' saved [130616/130616]
$ tar xzf rabbitmq-server-1.6.0.tar.gz
$ cd rabbitmq-server-1.6.0
$ make run
Copyright (C) 2007-2009 LShift Ltd., Cohesive Financial Technologies LLC., and Rabbit Technologies Ltd.
Licensed under the MPL. See http://www.rabbitmq.com/
node : rabbit@crescent
log : /tmp/rabbit.log
sasl log : /tmp/rabbit-sasl.log
database dir: /tmp/rabbitmq-rabbit-mnesia
starting database ...done
starting core processes ...done
starting recovery ...done
starting persister ...done
starting guid generator ...done
starting builtin applications ...done
starting TCP listeners ...done
broker running
RabbitMQ is now listening on localhost port 5672. Leave this terminal open and switch to a new one.
3) Get QFeedreader/Minion
Fetch a copy of qfeedreader-minion:
$ git clone git://github.com/adamwiggins/qfeedreader-minion.git
4) Rails Setup
Run the standard Rails app setup:
$ rake db:migrate
$ script/server
Leave that terminal open, there’s one more process to run.
5) Run a Worker
With Delayed Job, this is where you would run “rake jobs:work”. In the Minion port of QFeedreader, we’ve created an arbitrary worker named worker.rb in the root directory. Run that directly:
$ ruby worker.rb
Tue Sep 22 12:38:02 -0700 2009 :minion: Starting minion
Tue Sep 22 12:38:02 -0700 2009 :minion: subscribing to feed.fetch
With these three processes running (RabbitMQ, web process, worker process), we’re ready to test it out. Go to http://localhost:3000 and enter a URL, such as http://adam.blog.heroku.com/, and click Add. Now flip to the terminal running the worker process and you should see this:
Tue Sep 22 12:39:24 -0700 2009 :minion: recv: feed.fetch:{"url": "http://adam.blog.heroku.com/"}
Minion logs that it has received a message named feed.fetch, with the url parameter equal to what we typed in. It processed the job without errors, and flipping back to the browser, we can click reload to see the fetched feed.
Scale to the Sky
We’ve been using RabbitMQ as the backbone of the Heroku production system for over a year. Rabbit has had a few growing pains along the way, but as of version 1.6 it’s become a mature and capable message bus for anyone looking to do high-volume, low-latency job queueing.
Minion puts an simple and elegant face on the otherwise nearly inscrutable AMQP protocol and eventmachine implementation, removing a major barrier for Rubyists to be able to use this pro-grade messaging system.
The Minion worker process is single-threaded (which is good, because threads suck). To scale out your background job capacity, run more instances of worker.rb. Your web processes and worker processes can be scaled independently according to the need of your app. You can run them on the same machine or split them across different machines as you prefer.
Congratulations, you’re now queueing with the big boys.