Capistrano Off the Beaten Path

Introduction

If you use Capistrano, most likely you use it to deploy rails applications by running it from the project directory. The plugem management tool piggy-backs on Capistrano to execute some recipes without using the current path because recipes operate on gems. Recipes do not, however, completely ignore the current directory, but instead use it for optional customization. Customization is not limited to a deployment recipe from the current directory but can be environment-specific across multiple sites if they each have a special extension gem installed.

Background

We use a bunch of ruby scripts at RHG for deployment and delivery of our numerous applications and shared component. When we started preparing some of that functionality for a public release, we needed to provide a way to customize the scripts. Warren Konkel suggested to migrate them to Capistrano since it was the most familiar tool for rails developers and it has a recipes hierarchy that can be used for customization. So, we wrote the plugem command tool that feeds its parameters to Capistrano via Capistrano::CLI.new(plugem_converted_arguments).execute!
after loading its own recipes.

The Code

    1 module Capistrano
    2   class Configuration
    3
    4     alias :standard_cap_load :load
    5
    6     def load(*args, &block)
    7
    8       standard_cap_load(*args, &block)
    9      
   10       if args == ["standard"]
   11
   12         load_plugem_deploy_recipes(File.dirname(__FILE__) + '/../..')
   13
   14         begin
   15           require 'plugems_deploy_ext'
   16           load_plugem_deploy_recipes(PLUGEMS_DEPLOY_EXT_DIR) # Overriding from extensions
   17         rescue Exception
   18           # No extension is loaded
   19         end
   20
   21       end
   22
   23     end
   24    
   25     def load_plugem_deploy_recipes(dir)
   26       Dir[File.join(dir, 'recipes', '*')].each { |f| standard_cap_load(f) }
   27     end
   28      
   29   end
   30 end

Ignoring lines 14-19 for now, all it does is inject the plugem deployment gem recipes to Capistrano. They are now available for execution by Capistrano. So when I call plugem update my_app it is being translated to cap plugem_update -s plugem_name=my_app(we use the plugem_ namespace for tasks and variables to avoid clashing with standard recipes). Since loading of plugem recipes is purposely not handled via the -f flag of capistrano, the standard deployment recipes like config/deploy.rb are still being loaded.

Customization

The package is extending Capistrano but we wanted it to be customized too. For example, you might want to define your own list of gem servers to download gems from. Capistrano allows to define recipes per project, user, or host. The plugems_deploy takes it a step further and provides a way to customize it per deployment environment, which might contain many hosts. It does this by expecting an optional plugems_deploy_ext gem to be installed on a system (lines 14-19). If it finds the extension gem, it loads recipes from there overriding the default ones.


Conclusion

Capistrano is not only a great deployment tool, it can be used as a base of highly-customizable general purpose tools.

Plugems - Why views in plugems?

When we started to develop Plugems we had multiple applications that all shared a similar look and feel. One concept that emerged recently for me was that layouts can act like objects. In our suite of applications, we have a base layout that each application can derive from to satisfy it's unique set of presentation requirements. For example our UI plugem has the following:

 ui/views/layouts/default.rhtml

Using a little magic from fora.pragprog we can extend our default layout writing the following:

<% inside_layout 'default' do -%>
     <%= @content_for_layout %>
  <% end -%>

As well as with layouts, there are also smaller blocks of UI that make up a page that can be shared. I've always thought of these as widgets, which in many cases can be defined as a set of partials and helpers. An application may place a widget on a view, feeding it the data needed to bring it to life. This is different from components because there is no controller logic. Really the presentation side of a widget can be expressed by a set of partials, Javascript, CSS and a healthy dose of ruby [e.g. helpers]. In this way partials are the basic building block on which many UI components can be built. By themselves they are incomplete, but within the context of an application they are alive...

Plugems enables you to share common UI widgets across multiple applications. Common assets (css, js, images) can live in a plugem, but be shared across applications at deployment time. I hope to post more in depth on this topic in near future.

DRYing Up Configuration Files

Introduction

Our post about deployment process explains that some configuration files are overridden at the time of deployment. Maintaining those files could quickly become a nightmare unless the development team constantly evaluates them to gather only pieces that differ into the deployment specific configs, i.e. keeps them DRY. It has been a long journey for us and we are still adjusting the config file usage for our project.


ConfigurationLoader

We access all configuration files from our code via ConfigurationLoader (and, yes, it is a plugem itself). It provides some convenience methods for major configs. For example, to establish a non-default connection from a model, this piece of code could be used:

establish_connection ConfigLoader.load_db_config['secondary_db']

The load_db_config method knows that database.yml is sectioned per environment so it loads the file and returns the corresponding to current RAILS_ENV section. ConfigLoader is an instance of ConfigurationLoader with caching enabled. Since we use a lot of configuration-driven parameters in our models and controllers, caching this info saves unnecessary file system calls and ERB parsing.


DRYing Up By Keeping Configs Close to The Source

Another property of ConfigurationLoader is that it looks not only in the application config directory but in configs of plugins and gems. All found configs are merged (for usual config ruby structures like Hash and Array) in the order gems<-plugins<-application. It allows us to keep the default configuration within a gem to be shared between multiple applications, and, at the same time, lets applications overwrite it if needed. Together with the power of deployment time overriding, which is applicable for shared components as well, it helps us to keep only application-specific entries in its config files. The following is a concrete example from today: We have been using the Browser Logger as a plugin in our applications. Aaron decided to convert it into a plugem. We obviously don't want it to be enabled in our production deployment environment but we want to be able to see logs through a browser on developers' workstations and in QA. The gem initializer loads browser_loggger.yml via ConfigurationLoader to determine whether it is enabled or not. It is not DRY to put browser_loggger.yml in every application that uses Browser Logger. Instead, the gem itself contains config/browser_loggger.yml, with the property enabled, config/deployment/prod-browser_loggger.yml, with the property disabled, and config/deployment.yml with a single entry 'files: browser_loggger.yml' which defines that config/browser_loggger.yml will be overridden on production boxes with the content of the second file (see When Capistrano Is Not Enough for details how deployment time configuration works).


DRYing Up By Extracting Changing Parts in A Separate File

Our rails applications call a lot of backend services. Services configuration is stored in the service.yml file. Service endpoints usually follow same naming conventions in respect to a port and a path between different deployment environments. So at some point, we extracted host names into a separate file (service-hosts.yml) and changed our deployment.yml to override it instead of service.yml at the deployment time. Since service.yml is ERB-processed, we leverage it for loading the host file:

<% hosts = ConfigLoader.load_file('service-hosts.yml') %>

service_cfg: &service_cfg

foo:
url: http://<%= hosts['foo'] %>:8080/services/foo
timeout: 5

bar:
url: http://<%= hosts['bar'] %>:8080/services/bar
timeout: 5

production: *service_cfg
development: *service_cfg
test: *service_cfg

Conclusion

Our experience has shown that combination of deployment and run time configurations is powerful enough to handle most use cases we have had so far. There is always room for improvement, however. For example, we have lately run into a problem when one of the production boxes needed one of parameters in service.yml set to a different value then the rest of production boxes. We can address it within the current framework by copying a content of prod-sevice-hosts.yml to prod-box-1-service-hosts.yml and changing the parameter there. Since the host name (prod-box-1) takes precedences over the host class name (prod) it would fix the issue but the maintenance cost of two almost identical files is high. An alternative solution would be to enhance the framework to allow additional runtime overriding of values of service.yml from service-override.yml. Then we can create prod-box-1-service-override.yml which would contain a single entry with the parameter it needs to override. Work in progress...

Locking down a deployed application

A single host might have multiple applications installed each on a different release track. Since all our applications are distributed as a bunch of gems, there is a potential issue with shared components. Even though the whole portal is being tested when some application or component is being pushed through the stack, there is always a possibility of incompatibility of on of the upgraded shared component with another dependent application. In some other environments there would only one solution - to roll back the whole new application. In our environment we have another option - to lock down the affected application. When an application is deployed, one of the deployment steps is to create a gem repository inside the application structure. The application gem repository has the sources gem installed and the rest of gems sym-linked as in this example:

actionmailer-1.2.5 -> /usr/lib/ruby/gems/1.8/gems/actionmailer-1.2.5
actionpack-1.12.5 -> /usr/lib/ruby/gems/1.8/gems/actionpack-1.12.5
...
sources-0.0.1

Gems put into an internal gem structure are those it was tested with, so they are guarantee to work. All is needed to lock down an application now is to set the GEM_HOME environment variable on a web server instance, serving the application, and to point it to the application internal gem repository. After that the application won't use the latest versions of the gems but those specific it is locked to.

Plugems

Part of the Gem-based Development Process series

Implementations:

Runtime

Deployment

*****

We started with a small application. A few controllers, a few models. Soon the company grew, the development team grew, and with that the application grew. Luckily, we namespaced applications early on, so it was mostly clear where one application lived from another, but the rails project became too big. As one coworker (Eddie) put it, "My TextMate file find is getting too slow!". It was time to split applications. But for a portal with a single look and feel, how do you share things across applications? Here is how we do it (this week).

We wanted to use gems, as described in Part I, but without gems being true 1st class citizens in rails, there was no way to make it happen. So what was important to share and more importantly version?

I. Code
Thats obvious, rubygems already does (most of) this for us.

II. Views
With a common header, or commonly shared assets, we'd like to leverage those.

III. Rake Tasks
We have lots of rake tasks to do all sorts of things, we need to share those too.

IV. Assets
We'll cover this later. This gets complicated, and needs deployment input.

V. Plugems as Plugins
Using gems, as plugins, in your plugin dir, and the A/B/C problem.




I. Code
We didnt do (much of) anything to get this. Thanks guys who wrote RubyGems!

II. Views
We needed to load views from a few places, and it took some diving into rails internals to figure out how. There were a couple things we needed to figure out.

1. Where to look for views.
2. The order in which we look.
3. Caching. Globbing for files is slow... especially across many directories. Cant do this a lot.

1. So we first overwrote the ActionView::Base's implementation of finding files. That was straightforward. We also fixed ActionController::Layout to allow layouts to live in gems as well. Once we provided ActionView and ActionController with a bigger list of partials, it could then load the partial from anywhere. Now you'll see the path to a file in a gem in certain stack traces, just to prove that where to find it.

2. So why is the order important and powerful? Lets say you decide to use our base_ui gem and its default layout.


<%= render :partial => "header"%>
<%= @content_for_layout -%>
<%= render :partial => "footer"%>



In our gem,
/gems/base_ui/
/views/layouts/
default.rhtml
_header.rhtml
_footer.rhtml

You've been told to build the FooBar application, whos pages dont have the same header. So in your application, add a
/app/views/layouts/_header.rhtml, and put whatever you want there. It just works.

So the dependency order is: app ==> plugins ==> gems. There is almost always true. I'll explain the exception later when "developing" things that are gems.

Note: The compiled template method names get kinda ugly, but who cares.. You only see those in some stack traces.

3. Caching was quickly important. Disclaimer: We only look for views in gems and plugins at startup. After that, bounce your app to pick up new files. Thanks to Zed, mongrel_rails restart is wicked fast.

III. Gem Rake Tasks
So the easiest way to include rake tasks from gems was to add a bootstrap.rake into /lib/tasks/bookstrap. We grab the manifest file for the application, and for each of the gems defined, 'load' the rake files. This was a quick fix and could probably be more elegant.

IV. Assets
This conversation gets complicated quick. We'll cover this later.

V. Plugems ( Plugin + Gem = Plugem!)
So now you've taken all your favorite plugins and made them gems. In the next article, I'll cover developing gems as plugins, and the A --> B --> C dependency loading problem, where A and C are gems, and B is a plugin.

Gem-based Rails Dependency Loading: Part 0 - Introduction

Part of the Gem-based Development Process series

*****

Once you begin to build a lot of plugins (shared across numerous Rails applications), cross-plugin dependencies start to creep in. If you've ever renamed your plugins to effect load order, then you know what I'm talking about. Our idea, which has been extremely successful, is to package interesting dependencies as RubyGems. These dependencies include proprietary shared plugins as well as library dependencies.

Since our applications care to share a lot more than bundles of Ruby classes, we decided to beef up plugins as well.

• Part I: Justification
• Part II: Bundling a plugin as a Gem and getting it all to work with Rails
• Part III: Extracting views, partials, tasks and layouts into plugins and gems
• Part IV: Retaining developer convenience in a Gem-driven Rails application

Next time: Our justification behind this approach