Friday, March 09, 2007

DRYing Up Configuration Files


Our post about deployment process explains that some configuration files are overridden at the time of deployment. Maintaining those files could quickly become a nightmare unless the development team constantly evaluates them to gather only pieces that differ into the deployment specific configs, i.e. keeps them DRY. It has been a long journey for us and we are still adjusting the config file usage for our project.


We access all configuration files from our code via ConfigurationLoader (and, yes, it is a plugem itself). It provides some convenience methods for major configs. For example, to establish a non-default connection from a model, this piece of code could be used:

establish_connection ConfigLoader.load_db_config['secondary_db']

The load_db_config method knows that database.yml is sectioned per environment so it loads the file and returns the corresponding to current RAILS_ENV section. ConfigLoader is an instance of ConfigurationLoader with caching enabled. Since we use a lot of configuration-driven parameters in our models and controllers, caching this info saves unnecessary file system calls and ERB parsing.

DRYing Up By Keeping Configs Close to The Source

Another property of ConfigurationLoader is that it looks not only in the application config directory but in configs of plugins and gems. All found configs are merged (for usual config ruby structures like Hash and Array) in the order gems<-plugins<-application. It allows us to keep the default configuration within a gem to be shared between multiple applications, and, at the same time, lets applications overwrite it if needed. Together with the power of deployment time overriding, which is applicable for shared components as well, it helps us to keep only application-specific entries in its config files. The following is a concrete example from today: We have been using the Browser Logger as a plugin in our applications. Aaron decided to convert it into a plugem. We obviously don't want it to be enabled in our production deployment environment but we want to be able to see logs through a browser on developers' workstations and in QA. The gem initializer loads browser_loggger.yml via ConfigurationLoader to determine whether it is enabled or not. It is not DRY to put browser_loggger.yml in every application that uses Browser Logger. Instead, the gem itself contains config/browser_loggger.yml, with the property enabled, config/deployment/prod-browser_loggger.yml, with the property disabled, and config/deployment.yml with a single entry 'files: browser_loggger.yml' which defines that config/browser_loggger.yml will be overridden on production boxes with the content of the second file (see When Capistrano Is Not Enough for details how deployment time configuration works).

DRYing Up By Extracting Changing Parts in A Separate File

Our rails applications call a lot of backend services. Services configuration is stored in the service.yml file. Service endpoints usually follow same naming conventions in respect to a port and a path between different deployment environments. So at some point, we extracted host names into a separate file (service-hosts.yml) and changed our deployment.yml to override it instead of service.yml at the deployment time. Since service.yml is ERB-processed, we leverage it for loading the host file:

<% hosts = ConfigLoader.load_file('service-hosts.yml') %>

service_cfg: &service_cfg

url: http://<%= hosts['foo'] %>:8080/services/foo
timeout: 5

url: http://<%= hosts['bar'] %>:8080/services/bar
timeout: 5

production: *service_cfg
development: *service_cfg
test: *service_cfg


Our experience has shown that combination of deployment and run time configurations is powerful enough to handle most use cases we have had so far. There is always room for improvement, however. For example, we have lately run into a problem when one of the production boxes needed one of parameters in service.yml set to a different value then the rest of production boxes. We can address it within the current framework by copying a content of prod-sevice-hosts.yml to prod-box-1-service-hosts.yml and changing the parameter there. Since the host name (prod-box-1) takes precedences over the host class name (prod) it would fix the issue but the maintenance cost of two almost identical files is high. An alternative solution would be to enhance the framework to allow additional runtime overriding of values of service.yml from service-override.yml. Then we can create prod-box-1-service-override.yml which would contain a single entry with the parameter it needs to override. Work in progress...

Wednesday, March 07, 2007

Gem-based Deployment and Delivery: Part 2 - Distribution via Gem-servers

Part of the Gem-based Development Process series:

The Idea

Once we packaged all of our applications and shared components as gems, it did not take us long to realize that we could leverage the native method of gem distribution - gem servers - to build an infrastructure for pushing products through the dev/qa/production deployment stack. The idea of gem promotion was born.

There are three layers of gem servers - one per deployment environment (dev/qa/production). Gems can be promoted from the upstream gem server to the downstream one. The direction of promotion is dev->qa->prod. The individual boxes within the deployment environment install gems only from the server that serves their environment. For example, when a QA person installs gems on qa7-rails & qa8-rails boxes, he uses the qa-gems server as the source (as in gem install xyz --source qa-gems --remote). The dev gem server is the base one where freshly built gems are pushed to.

The Implementation

Since our QA and Operations teams wanted to have full control over what they have on their gem servers, we adopted the push-pull model instead of a more simple push-only one (when a gem promotion works by pushing from the upstream to the downstream). In our current model, developers push gems to the base (dev) gem server first. After it is tested and is QA ready, the manifest file, containing a specific gem name and version (together with names/version of all dependent gems), is generated and sent to QA. A QA installer uses the manifest to pull gems from the upstream dev gem server to the downstream QA one. He then goes to the individual QA boxes and upgrades the gems there. When QA clears the build, the manifest is sent to Operations to repeat the procedure in production with the qa gem server being the upstream one.

We greatly simplified the usage by hiding decisions about upstream/downstream, the source gem, etc. behind a single tool that was deployment environment aware. For example, to pull down gems based on the manifest to the qa gem server, the installer issues a command rhg pull manifest_file on the qa gem server box. The rhg tool queries the hostname of the machine it is being run on, and based on its name (qaX-YYY), it picks the upstream server (dev-gem). Running rhg up gem_name gets the gem (and all of its dependencies) from the environment specific gem server, with the base (dev) gem server being the default option so developers can use the same command for updating their local gem repositories.

Under this scheme, each environment has full control over which gems are installed there and the only piece of information needed for promotion is the manifest file declaring specific versions to promote.

The Reality

We built the tools and dev/qa gem servers and started pushing gems to QA using the described approach. When the time came to implement it in production, our operations team rejected it. They had a reason: they wanted a unified approach for delivering all packages they need to install. Since we have Java and many other non-ruby applications while the ops requirement was to use RPMs for package distribution, we had to adapt. We retained the dev gem sever and continued to allow developers to push gems to it. We continue to generate the manifest file, but instead of delivering it as is, we use it to pick up the specific versions of gems and repackage them as RPMs (using a modified version of gem2rpm). They are then delivered to QA and later to production.

The Future

We still believe that using the 'gem servers' hierarchy is the right way for plugem-based applications distribution. Most likely the original scripts will be a part of the near future plugem public release. We hope that one day it might become the preferred method of rails applications distribution.

Tuesday, March 06, 2007

Gem-based Deployment and Delivery: Part 1 - When Capistrano Is Not Enough

Part of the Gem-based Development Process series


In the beginning, we, as many other rails projects, were using Capistrano for deployment. This changed when it came time to start preparing builds for QA and production. Having formal QA and Operations teams, we had to adjust our approach to meet their requirements for deployment. Another force pushing us off Capistrano--even for deployment in the development environment--was the existence of multiple servers where different versions of the application could be deployed by anyone from the large development team. As a result, we built our own deployment tools to be used by QA/Operations (and occasionally the development team.) We also use push-button builds (via luntbuild) to deploy applications to dev.

The Tools

A New RHG Developer's Illustrated Primer provides insight into how deployment tools are used on our projects. Since it stops at the point when the build is ready for QA, it does not show all use cases; however, it gives up enough to see what might be going on later in QA and production. The teams there use rhgcontrol for any application or shared component deployment task. Wrapping up all deployment activities in a self-contained tool gives the development team better control over how applications are installed and deployed in those environments. In addition, it allows the addition of new features to the existing command set without changing the installation instructions.

A couple of examples:

  1. When our DBAs asked us to provide an automated way to update the application history table with the current application upon deployment, we added auto-generation of a db migration that does just that to the rhgcontrol migrate command.
  2. We want to know what versions of shared components are used by the running version of an application. Instead of building a runtime configuration querying system a-la JMX, we simply fixed rhgcontrol start to dump its runtime configuration right after the application was started.

The Deployment

Any application or shared component is potentially deployable. All that is required is a single file--deployment.yml--in the config directory of the gem (since all applications at RHG are plugems). The content of the file is read and executed at deployment time when rhg deploy is run. The deployment configuration is simpler than a Capistrano recipe and contains just a few sections.

Sample deployment.yml:

# For servers
server: &server_cfg

- database.yml
- log4r.xml

- ln -nfs <%= @app_home %> <%= @app_install_dir %>/<%= @app_name %>
- <%= @cmd_host_gems %>
- rake some:action

qa: *server_cfg
dev: *server_cfg

# For developer's workstations

- database.yml

- <%= @cmd_link_to_app %>

First, there is a separation by a deployment environment - dev/qa/etc. Those names are based on the hostname of a machine where rhg deploy is executed. We adopted unified DNS naming conventions for our rails boxes (e.g. all of our QA machine names start with qa.) This allows us to define host classes and put, say, QA-specific tasks. If the hostname does not match any class, the default section is used.

There are two types of the deployment tasks: copying over configuration files (the files subsection) and execution of commands (the execs section).

Some configuration files, such as database.yml, are often deployment environment specific. The deployment-time configuration directory (config/deployment) may contain such files, prefixed with either the host class (like dev) or the full hostname (like dev8-rails.) They are used to overwrite the copies under the config/ directory at deployment time.

Configuration-file overrides are a small piece of the multiple-configuration puzzle we've had to solve at RHG. We plan to have a separate post on this subject, which will cover runtime configuration as well.

There are two types of the execution commands--regular UNIX commands and macros. Macros are commands that are bundled with the rhg tool (when it makes sense to share them between different applications.) For example, the macros for hosting gems (@cmd_hosts_gems) looks like this:

- rm -rf gems
- tar -xf <%= @deployment_gem_dir %>/templates/gems-bare.tar

Instead of putting those commands in the deployment.yml file of every application that uses the locking mechanism, we share them via macros. They also provide us a single place to change the implementation of these common tasks.

The execs section leverages deployment-specific variables, resulting in a system flexible enough to handle every deployment task we've encountered.


While Capistrano is suitable for most of the rails projects out there, we had to build our own deployment and delivery tools to wrangle our multi-environment, multi-application, multi-component portal. We plan to extract and release the useful parts of the tool for Rails development teams that are isolated from their QA/Production environments.