Incremental - Always

methodology rush

Fri Mar 07 01:12:00 -0800 2008

Programmers generally agree that working iteratively is a good idea. But sometimes, we’ll say: you just can’t. This particular problem has to be done in one big bite; there’s no way to break it down into smaller pieces; we just have to take the plunge.

Poppycock, I say. (Or perhaps something stronger.)

I’ve begun the process of using rush in Heroku’s infrastructure. But this is tough: it’s an unproven and immature library, barely a month old. Worse, most of the things we want it for - system-level calls - are of critical importance. Misplaced files are not forgiveable the way that, say, a UI glitch might be.

I started by selecting the absolute least important bit of code it could be used for. This component is something that happens very infrequently: rsyncing config files when launching a particular kind of instance. Further, I launch these manually (unlike some other instance types which self-scale), so there will always be someone watching the logs when it boots up.

But I took it even a step further. I kept the old method around and, in the case of exception on the newly rushified method, call it as a fallback. Here’s the code:

  def sync_nginx_conf
    sync_nginx_conf_via_rush
  rescue Exception => e
    Log.error "Exception on sync_nginx_conf via rush, falling back to rsync: #{e.summary}", :addendum => e.full_display
    sync_nginx_conf_via_rsync
  end

This is not foolproof; something in the rush sync could write an incorrect file without throwing an exception, for example. I’ll keeping a very close eye on this method when it runs in production. Once this has proven itself in action, I’ll feel confident in taking a bigger step.