Gem-based Deployment and Delivery: Part 1 - When Capistrano Is Not Enough

Part of the Gem-based Development Process series

Introduction

In the beginning, we, as many other rails projects, were using Capistrano for deployment. This changed when it came time to start preparing builds for QA and production. Having formal QA and Operations teams, we had to adjust our approach to meet their requirements for deployment. Another force pushing us off Capistrano--even for deployment in the development environment--was the existence of multiple servers where different versions of the application could be deployed by anyone from the large development team. As a result, we built our own deployment tools to be used by QA/Operations (and occasionally the development team.) We also use push-button builds (via luntbuild) to deploy applications to dev.


The Tools

A New RHG Developer's Illustrated Primer provides insight into how deployment tools are used on our projects. Since it stops at the point when the build is ready for QA, it does not show all use cases; however, it gives up enough to see what might be going on later in QA and production. The teams there use rhgcontrol for any application or shared component deployment task. Wrapping up all deployment activities in a self-contained tool gives the development team better control over how applications are installed and deployed in those environments. In addition, it allows the addition of new features to the existing command set without changing the installation instructions.

A couple of examples:

1. When our DBAs asked us to provide an automated way to update the application history table with the current application upon deployment, we added auto-generation of a db migration that does just that to the rhgcontrol migrate command.
2. We want to know what versions of shared components are used by the running version of an application. Instead of building a runtime configuration querying system a-la JMX, we simply fixed rhgcontrol start to dump its runtime configuration right after the application was started.

The Deployment

Any application or shared component is potentially deployable. All that is required is a single file--deployment.yml--in the config directory of the gem (since all applications at RHG are plugems). The content of the file is read and executed at deployment time when rhg deploy is run. The deployment configuration is simpler than a Capistrano recipe and contains just a few sections.

Sample deployment.yml:

# For servers
server: &server_cfg

files:
- database.yml
- log4r.xml

execs:
- ln -nfs <%= @app_home %> <%= @app_install_dir %>/<%= @app_name %>
- <%= @cmd_host_gems %>
- rake some:action


qa: *server_cfg
dev: *server_cfg

# For developer's workstations
default:

files:
- database.yml

execs:
- <%= @cmd_link_to_app %>

First, there is a separation by a deployment environment - dev/qa/etc. Those names are based on the hostname of a machine where rhg deploy is executed. We adopted unified DNS naming conventions for our rails boxes (e.g. all of our QA machine names start with qa.) This allows us to define host classes and put, say, QA-specific tasks. If the hostname does not match any class, the default section is used.

There are two types of the deployment tasks: copying over configuration files (the files subsection) and execution of commands (the execs section).

Some configuration files, such as database.yml, are often deployment environment specific. The deployment-time configuration directory (config/deployment) may contain such files, prefixed with either the host class (like dev) or the full hostname (like dev8-rails.) They are used to overwrite the copies under the config/ directory at deployment time.

Configuration-file overrides are a small piece of the multiple-configuration puzzle we've had to solve at RHG. We plan to have a separate post on this subject, which will cover runtime configuration as well.

There are two types of the execution commands--regular UNIX commands and macros. Macros are commands that are bundled with the rhg tool (when it makes sense to share them between different applications.) For example, the macros for hosting gems (@cmd_hosts_gems) looks like this:

- rm -rf gems
- tar -xf <%= @deployment_gem_dir %>/templates/gems-bare.tar

Instead of putting those commands in the deployment.yml file of every application that uses the locking mechanism, we share them via macros. They also provide us a single place to change the implementation of these common tasks.

The execs section leverages deployment-specific variables, resulting in a system flexible enough to handle every deployment task we've encountered.


Conclusion

While Capistrano is suitable for most of the rails projects out there, we had to build our own deployment and delivery tools to wrangle our multi-environment, multi-application, multi-component portal. We plan to extract and release the useful parts of the tool for Rails development teams that are isolated from their QA/Production environments.