Showing posts with label rhg. Show all posts
Showing posts with label rhg. Show all posts

Sunday, June 17, 2007

Code Digest #1

When you program for a living, you write lots of code. There is often some code that you are fond of. We start the Code Digest series to present such code written by the RHG developers. We encourage other teams and individual developers to share similar snippets in their blogs so we all can learn from each other and become better rails developers.

Daniel Silva



<%= link_to(image_tag("/images/icon.gif", :style=>"padding-right:0px;"), {:controller => '/registration', :action => 'login', :dest => request.request_uri}, {:style => ""}) %> 

With this line, I can create an image link using Rails code, to take advantage of the power of routes in Rails while still creating this kind of structure: <a href="#"><img border="0"/></a>



Jack Dempsey



Ok, so here's something useful for using keyboard short cuts to navigate through a small supporting application:
def include_keyboard_shortcuts

hotkey = 'Ctrl+Shift'
code_string = ''
keys = {
'm'=>'/account_info/show',
's'=>'/account_info/index',
'c'=>'/menu/go?menu_action=new_customer',
'p'=>'/products',
'o'=>'/account_info/orders',
't'=>'/order_tracker',
'x'=>'/auth_sessions/destroy',
'r'=>'/current_users/reset_password'
}

current_user_needed = %w{m o r}

keys.each_pair do |k,v|
next if current_user_needed.include?(k) && current_user.nil?
code_string << "shortcut('#{hotkey}+hm#{k}',function() { document.location.href='#{v}' });\n"
end

javascript_tag(code_string)

end



Then in a layout file just:
<%= javascript_include_tag 'shortcuts' %>
<%= include_keyboard_shortcuts %>



Makes use of a nice shortcuts.js lib.


Val Aleksenko


This is a fragment of a conversation on our internal IRC with my solution for refactoring a piece of code:
May 07 14:07:56 <Anthro> There has to be a better way to do this (x and y are non-negative integers): increment = (x!=0) ? ( (y!=0) ? 0 : 1 ) : -1
May 07 14:08:22 <Anthro> I think it can be done with math instead of logic.
May 07 14:16:56 <jack> Anthro: have you thought about using sin and cos? ;-)
May 07 14:18:42 <muzzy> (x <=> 0) <=> (y <=> 0)



Granted, it is not as clear as the original, but it is hard to resist it from the aesthetic point of view.

Val Aleksenko


I love writing dynamically generated methods and classes in Ruby as the next guy. When I was writing acts_as_readonlyable, I needed to manage multiple connections. I found that the easiest solution for managing active connections was generation of an AR class for each read-only definition and borrowing the connection from it. The usage is acts_as_readonlyable :read_only_entry_in_database_config.
def acts_as_readonlyable(readonly_db)
define_readonly_class(readonly_db) unless ActiveRecord.const_defined?(readonly_class_name(readonly_db))
end

def readonly_class_name(db)
"Generated#{ db.camelize }"
end

def define_readonly_class(db)
ActiveRecord.module_eval %Q!
class #{ readonly_class_name(db) } < Base
self.abstract_class = true
establish_connection configurations[RAILS_ENV]['#{ db }']
end
!

end



Warren Konkel


The ConfigFile class is a quick and easy way to load YAML files located in your Rails application's config directory. Simply place a YAML file into your config directory and then access it like a hash. For example this will read your config/database.yml: ConfigFile['database']['production']['adapter'].
class ConfigFile
def self.[](arg)
@@cached_configs ||= {}
@@cached_config_mtimes ||= {}

base_name = "config/#{arg}.yml"
filename = File.join(RAILS_ROOT, base_name)
raise "ERROR: Config not found: #{base_name}" unless File.exists?(base_name)

if @@cached_configs[arg].nil? || @@cached_config_mtimes[arg] < File.stat(filename).mtime.to_i
@@cached_configs[arg] = YAML.load(File.open(filename))
@@cached_config_mtimes[arg] = File.stat(filename).mtime.to_i
end

@@cached_configs[arg]
end
end

Monday, May 14, 2007

Capistrano Off the Beaten Path

Introduction

If you use Capistrano, most likely you use it to deploy rails applications by running it from the project directory. The plugem management tool piggy-backs on Capistrano to execute some recipes without using the current path because recipes operate on gems. Recipes do not, however, completely ignore the current directory, but instead use it for optional customization. Customization is not limited to a deployment recipe from the current directory but can be environment-specific across multiple sites if they each have a special extension gem installed.

Background

We use a bunch of ruby scripts at RHG for deployment and delivery of our numerous applications and shared component. When we started preparing some of that functionality for a public release, we needed to provide a way to customize the scripts. Warren Konkel suggested to migrate them to Capistrano since it was the most familiar tool for rails developers and it has a recipes hierarchy that can be used for customization. So, we wrote the plugem command tool that feeds its parameters to Capistrano via Capistrano::CLI.new(plugem_converted_arguments).execute!
after loading its own recipes.

The Code

    1 module Capistrano
2 class Configuration
3
4 alias :standard_cap_load :load
5
6 def load(*args, &block)
7
8 standard_cap_load(*args, &block)
9
10 if args == ["standard"]
11
12 load_plugem_deploy_recipes(File.dirname(__FILE__) + '/../..')
13
14 begin
15 require 'plugems_deploy_ext'
16 load_plugem_deploy_recipes(PLUGEMS_DEPLOY_EXT_DIR) # Overriding from extensions
17 rescue Exception
18 # No extension is loaded
19 end
20
21 end
22
23 end
24
25 def load_plugem_deploy_recipes(dir)
26 Dir[File.join(dir, 'recipes', '*')].each { |f| standard_cap_load(f) }
27 end
28
29 end
30 end

Ignoring lines 14-19 for now, all it does is inject the plugem deployment gem recipes to Capistrano. They are now available for execution by Capistrano. So when I call plugem update my_app it is being translated to cap plugem_update -s plugem_name=my_app(we use the plugem_ namespace for tasks and variables to avoid clashing with standard recipes). Since loading of plugem recipes is purposely not handled via the -f flag of capistrano, the standard deployment recipes like config/deploy.rb are still being loaded.

Customization

The package is extending Capistrano but we wanted it to be customized too. For example, you might want to define your own list of gem servers to download gems from. Capistrano allows to define recipes per project, user, or host. The plugems_deploy takes it a step further and provides a way to customize it per deployment environment, which might contain many hosts. It does this by expecting an optional plugems_deploy_ext gem to be installed on a system (lines 14-19). If it finds the extension gem, it loads recipes from there overriding the default ones.


Conclusion

Capistrano is not only a great deployment tool, it can be used as a base of highly-customizable general purpose tools.

Monday, April 23, 2007

[APPLICATION RELEASE] R::PasteBin

Introduction

Wikipedia pastebin definition:

A Pastebin, Also known as a nopaste, is a web application which allows its users to upload snippets of text, usually samples of source code, for public viewing. It is very popular in IRC channels where pasting large texts is considered bad etiquette.


R::Pastebin

We created this project because we needed a pastebin for internal usage. Being a rails shop, we wanted one any developer could enhance. We looked around for rails-based pastebins and even though there is a great Attachr available, we could not find anything to install internally. Since pastebins are trivial, we just implemented our own in no time, thanks to Rails and CodeRay. It is very basic but it works fine for us so we decided to release it.


Features

Code highlights for Ruby, RHTML, HTML, Plaintext, and C. The list of supported languages is likely to grow.

An optional irc notification (controlled by config/irc.yml)


Dependencies

rpastebin uses the coderay gem for syntax highlighting. The latest version can be installed via gem install coderay -rs http://rd.cYcnus.de/coderay

The IRC client needs the Ruby-IRC gem.

The rails code is 1.2.3 so if you have to stay on 1.1.6 (as we are), just rake rails:freeze:edge in the project directory.

The search feature utilizes fulltext so it is only compatible with the MySQL MyISAM engine.


Home

http://rubyforge.org/projects/rpastebin/


License

Distributed under MIT License


Thursday, April 19, 2007

Unveiling Revolution Pages

With the official launch, we're pleased to unveil the beta of codename "fabric"—the "Revolution Pages" platform. Our users will create 'long tail' topic-oriented pages that will help other users find resources on both the arcane and the mundane. The premise is that a page on 'Chemotherapy in Des Moines' could be inherently more interesting/useful than a page on 'Cancer'.

We believe that this application really shows off the capabilities of our home-grown Ruby portal server. Within our portal, most resources are implemented as 'modules' (and any un-modularized blocks of functionality are quickly being converted.) Fabric is simply a layer of client-side functionality (i.e. a boatload of javascript) that sits on top of our existing rendering/page composition capabilities.

Fabric provides the following capabilities:

1. The ability to 'clip' any module on the site into your private clipboard. Try it for yourself—register, then visit any page on the site (try healthy living or conditions for target-rich environments) and click on the scissors (in the upper right, with the page tools.) The modules that are clippable will illuminate.

Demo:






2. The ability to syndicate most modules on the site (the capability exists for all, but we don't syndicate everything for a variety of reasons.) Enter clipping mode and hit syndicate instead—you'll be given a chunk of javascript to paste into your website/blog/etc.

3. The ability to perform full-text search against every module that has ever been rendered on the portal (with a few exceptions where we explicitly choose not to ingest.) This means that not only can we surface pages based on full text, but we can also surface individual components of a page independently. We're using the power of SOLR via acts_as_solr (along with some stomp) for this, and it came out really clean. Props to the SOLR guys (expect some acts_as_solr enhancement patches + commentary on this topic shortly.)

Demo:







4. The ability to build pages using modules gathered via #1 and #3, with a few 'special' modules that bring in external resources (RSS feeds, rich text editor, image upload, YouTube, etc.) We even give you two means to lay out pages—columns and float. For the technical audience, think static vs. absolutely positioned. In float mode, you get Photoshop style snaplines; enable columns and you get 'straightjacketed' slots.

5. The ability to browse pages built via #4, either by full-text search or tag selection.


We developed some interesting client-side techniques to pull this off, and we'll be releasing some of the more generic stuff eventually.

Feel free to poke around and create a page for yourself:

http://www.revolutionhealth.com/fabric

Watch the full demo (still under development):








Recommended Browsers: FF 2.0, IE 7.0
Supported Browsers: Recommended browsers + IE 6
Potentially reasonable, but unsupported: Late versions of Opera, Webkit, Safari

RevolutionHealth Officially Launched!

Its been 3 months since we launched our preview site, and today I am proud to say we've launched the new and improved RevolutionHealth!

I look forward to our folks sharing some of the details behind our products in the near future.

Also, a big thank to the rails community. There is no way we could have done this without Rails, the community, and all the great plugins/blogs/articles out there.



Aside from a full redesign, we have some great new products such as:

KnowYourRisk - "Give us 10 minutes, we'll give you 10 years": A state-of-the-art assessment pioneered by BioSignia. In just 10 minutes, by confidentially entering basic health information, the KnowYourRisk(TM) proprietary program can calculate a user's risk of contracting one of the nine most common diseases – and can provide personalized information on actions to help reduce that risk and live longer.

Medicine Chest - A tool to find drugs and treatments, as well as rate and share your experience with others. This service is in its infancy, but our philosophy is that if millions of people participate, it will emerge as a useful tool for people as they consider their treatment options.



Revolution Pages - Now people can create and customize their own health pages, including their top-rated articles, tools and resources, as well as photos and personal stories. This sophisticated self-publishing capability encompasses Web 2.0's promise of creating customized health content: patients and families dealing with health issues can share the benefit of their experiences with others -- and learn from others, too. See some examples: Sneezing, Type II Diabetes, or Bad Knees.


These are along with many of the awesome products we launched in late January.

Thank you!

Saturday, March 31, 2007

[PLUGIN RELEASE] - Facade

Due to a slew of interesting runtime dependencies, we needed to be able to quickly mock out thorny pieces of our architecture.

  • I might not want to run a huge MOA stack to develop software that touches some arcane piece of our application that happens to use messaging.
  • I might want to hit a fully-functional mock (e.g. built in Active Record) instead of hitting live services that we don't own or might cost money.
  • I might want to crank out a naive implementation of a particular data source/sink with the intent of putting in a more 'industrial' solution if the product is a success.

Facade is a library for modeling objects that delegate some or all of their behavior to a surrogate class. It's essentially a DSL for describing a delegation arrangement, as well as a mechanism to bind a particular implementation to those delegator models. It's a little bit of dependency injection mixed with a little bit of IDL.

It also was a relatively enjoyable exercise in Ruby metaprogramming; this would have been quite painful in a few other languages.

Don't worry, it sounds much worse (and Java-esque) than it really is. Actual usage is pretty straightforward and doesn't create a lot of senseless abstraction.

Here's a little self-explanatory peek into the three pieces of the puzzle (the configuration, the facade model, and the implementation model(s).)

Facade model:

class Foo < Facade::Base
backend_class_method :delegated_class_method, :delegated_returns_a_collection_of_self
backend_instance_method :delegated_instance_method
def local_method
puts 'I do stuff that is indepenent of implmentation of delegated stuff'
end
end


Mock delegatee:

class Mock::Foo
def delegated_instance_method
puts 'I performed a mock implementation of my_instance_method'
end
def self.delegated_class_method
puts 'I performed a mock implementation of my_class_method'
end
def self.delegated_returns_a_collection_of_self
return [self.new, self.new]
end
end


Live delegatee:

class Live::Foo
def delegated_instance_method
puts 'I performed a live implementation of my_instance_method'
end
def self.delegated_class_method
puts 'I performed a live implementation of my_class_method'
end
def self.delegated_returns_a_collection_of_self
return [self.new, self.new]
end
end


Config file:

development:
foo:
backed_by: mock
production:
foo:
backed_by: live


The mock/live distinction isn't important--you could just as well have two mock implementations or three live implementations. It's just the name of the module that holds the implementation.

The unique thing about Facade (vis-a-vis other mocking solutions) is its autoboxing support. This allows delegatees to return instances of the delegator without even being aware that its a delegatee (in fact, the delegatee may be a class that you didn't even write.) Whenever instances of the delegatee are returned from a delegated method, an instance of the delegator is returned to the invoker instead. The reason we chose autoboxing over duck-typing is that you want to keep a bulk of the functionality in the implementation-agnostic facade model, and those methods would not be available if we simply returned references to the delegatee whenever the delegatee returned references of itself.

Example: If model Foo contains lots of AR-agnostic business logic but happens to delegate a couple of methods to its AR-backed delegatee--say, for persistence support--we want the results of those ActiveRecord finders to be instances of Foo < Facade::Base, not instances of ArModels::Foo < ActiveRecord::Base, or else we won't be able to invoke Foo's instance methods on the objects in the collection.

Here's a little console session that illustrates the usage of the models shown above:


epf-lap:/tmp/facade_hello_world $ ruby script/console
Loading development environment.
>> Foo
=> Foo
>> Foo.my_class_method
I performed a mock implementation of my_class_method
=> nil
>> foo = Foo.new
=> #<Foo:0x27bb584 @facade_backend=#<Mock::Foo:0x27bb55c>>
>> foo.my_instance_method
I performed a mock implementation of my_instance_method
=> nil

epf-lap:/tmp/facade_hello_world $ ruby script/console
Loading development environment.
>> foo = Foo.new
=> #<Foo:0x27c370c @facade_backend=#<Mock::Foo:0x27bf1c0>>
>> foo.delegated_instance_method
I performed a mock implementation of my_instance_method
=> nil
>> foo.local_method
I do stuff that is indepenent of implmentation of delegated stuff
=> nil
>> Foo.delegated_class_method
I performed a mock implementation of my_class_method
=> nil
>> Foo.delegated_returns_a_collection_of_self
=> [#<Foo:0x27ac64c @facade_backend=#<Mock::Foo:0x27ac728 @my_facade_model=#<Foo:0x27ac64c ...>>>, #<Foo:0x27ac50c @facade_backend=#<Mock::Foo:0x27ac714 @my_facade_model=#<Foo:0x27ac50c ...>>>]
>> ret_foo = Foo.delegated_returns_a_collection_of_self.first
=> #<Foo:0x279c454 @facade_backend=#<Mock::Foo:0x279cbe8 @my_facade_model=#<Foo:0x279c454 ...>>>
>> ret_foo.class
=> Foo
>> ret_foo.delegated_instance_method
I performed a mock implementation of my_instance_method
=> nil
>> ret_foo.local_method
I do stuff that is indepenent of implmentation of delegated stuff
=> nil


epf-lap:/tmp/facade_hello_world $ ruby script/console production
Loading production environment.
>> # notice it's using the other implementation now that we're in prod mode
>> Foo.delegated_class_method
I performed a live implementation of my_class_method
=> nil
>> # and you can dynamically change the implementation, complete with autobox
?> Foo.reset_backend(Mock::Foo)
=> Mock::Foo
>> Foo.delegated_class_method
I performed a mock implementation of my_class_method
=> nil
>> Foo.delegated_returns_a_collection_of_self.first.class
=> Foo
>> Foo.delegated_returns_a_collection_of_self.first.delegated_instance_method
I performed a mock implementation of my_instance_method


Check out the README for information on installation (it's simple!) and getting started.

Readme: svn cat \
svn://rubyforge.org/var/svn/facade/trunk/vendor/plugins/facade/README
Repository: svn checkout \
svn://rubyforge.org/var/svn/facade/trunk/vendor/plugins/facade
Rubyforge: http://rubyforge.org/projects/facade/

Installation:

ruby script/plugin install \
svn://rubyforge.org/var/svn/facade/trunk/vendor/plugins/facade

ruby script/plugin install \
svn://rubyforge.org/var/svn/config-loader/trunk/vendor/plugins/configuration_loader

Friday, March 09, 2007

DRYing Up Configuration Files

Introduction

Our post about deployment process explains that some configuration files are overridden at the time of deployment. Maintaining those files could quickly become a nightmare unless the development team constantly evaluates them to gather only pieces that differ into the deployment specific configs, i.e. keeps them DRY. It has been a long journey for us and we are still adjusting the config file usage for our project.


ConfigurationLoader

We access all configuration files from our code via ConfigurationLoader (and, yes, it is a plugem itself). It provides some convenience methods for major configs. For example, to establish a non-default connection from a model, this piece of code could be used:

establish_connection ConfigLoader.load_db_config['secondary_db']

The load_db_config method knows that database.yml is sectioned per environment so it loads the file and returns the corresponding to current RAILS_ENV section. ConfigLoader is an instance of ConfigurationLoader with caching enabled. Since we use a lot of configuration-driven parameters in our models and controllers, caching this info saves unnecessary file system calls and ERB parsing.


DRYing Up By Keeping Configs Close to The Source

Another property of ConfigurationLoader is that it looks not only in the application config directory but in configs of plugins and gems. All found configs are merged (for usual config ruby structures like Hash and Array) in the order gems<-plugins<-application. It allows us to keep the default configuration within a gem to be shared between multiple applications, and, at the same time, lets applications overwrite it if needed. Together with the power of deployment time overriding, which is applicable for shared components as well, it helps us to keep only application-specific entries in its config files. The following is a concrete example from today: We have been using the Browser Logger as a plugin in our applications. Aaron decided to convert it into a plugem. We obviously don't want it to be enabled in our production deployment environment but we want to be able to see logs through a browser on developers' workstations and in QA. The gem initializer loads browser_loggger.yml via ConfigurationLoader to determine whether it is enabled or not. It is not DRY to put browser_loggger.yml in every application that uses Browser Logger. Instead, the gem itself contains config/browser_loggger.yml, with the property enabled, config/deployment/prod-browser_loggger.yml, with the property disabled, and config/deployment.yml with a single entry 'files: browser_loggger.yml' which defines that config/browser_loggger.yml will be overridden on production boxes with the content of the second file (see When Capistrano Is Not Enough for details how deployment time configuration works).


DRYing Up By Extracting Changing Parts in A Separate File

Our rails applications call a lot of backend services. Services configuration is stored in the service.yml file. Service endpoints usually follow same naming conventions in respect to a port and a path between different deployment environments. So at some point, we extracted host names into a separate file (service-hosts.yml) and changed our deployment.yml to override it instead of service.yml at the deployment time. Since service.yml is ERB-processed, we leverage it for loading the host file:


<% hosts = ConfigLoader.load_file('service-hosts.yml') %>

service_cfg: &service_cfg

foo:
url: http://<%= hosts['foo'] %>:8080/services/foo
timeout: 5

bar:
url: http://<%= hosts['bar'] %>:8080/services/bar
timeout: 5

production: *service_cfg
development: *service_cfg
test: *service_cfg

Conclusion

Our experience has shown that combination of deployment and run time configurations is powerful enough to handle most use cases we have had so far. There is always room for improvement, however. For example, we have lately run into a problem when one of the production boxes needed one of parameters in service.yml set to a different value then the rest of production boxes. We can address it within the current framework by copying a content of prod-sevice-hosts.yml to prod-box-1-service-hosts.yml and changing the parameter there. Since the host name (prod-box-1) takes precedences over the host class name (prod) it would fix the issue but the maintenance cost of two almost identical files is high. An alternative solution would be to enhance the framework to allow additional runtime overriding of values of service.yml from service-override.yml. Then we can create prod-box-1-service-override.yml which would contain a single entry with the parameter it needs to override. Work in progress...

Wednesday, March 07, 2007

Gem-based Deployment and Delivery: Part 2 - Distribution via Gem-servers

Part of the Gem-based Development Process series:

The Idea

Once we packaged all of our applications and shared components as gems, it did not take us long to realize that we could leverage the native method of gem distribution - gem servers - to build an infrastructure for pushing products through the dev/qa/production deployment stack. The idea of gem promotion was born.

There are three layers of gem servers - one per deployment environment (dev/qa/production). Gems can be promoted from the upstream gem server to the downstream one. The direction of promotion is dev->qa->prod. The individual boxes within the deployment environment install gems only from the server that serves their environment. For example, when a QA person installs gems on qa7-rails & qa8-rails boxes, he uses the qa-gems server as the source (as in gem install xyz --source qa-gems --remote). The dev gem server is the base one where freshly built gems are pushed to.


The Implementation

Since our QA and Operations teams wanted to have full control over what they have on their gem servers, we adopted the push-pull model instead of a more simple push-only one (when a gem promotion works by pushing from the upstream to the downstream). In our current model, developers push gems to the base (dev) gem server first. After it is tested and is QA ready, the manifest file, containing a specific gem name and version (together with names/version of all dependent gems), is generated and sent to QA. A QA installer uses the manifest to pull gems from the upstream dev gem server to the downstream QA one. He then goes to the individual QA boxes and upgrades the gems there. When QA clears the build, the manifest is sent to Operations to repeat the procedure in production with the qa gem server being the upstream one.

We greatly simplified the usage by hiding decisions about upstream/downstream, the source gem, etc. behind a single tool that was deployment environment aware. For example, to pull down gems based on the manifest to the qa gem server, the installer issues a command rhg pull manifest_file on the qa gem server box. The rhg tool queries the hostname of the machine it is being run on, and based on its name (qaX-YYY), it picks the upstream server (dev-gem). Running rhg up gem_name gets the gem (and all of its dependencies) from the environment specific gem server, with the base (dev) gem server being the default option so developers can use the same command for updating their local gem repositories.

Under this scheme, each environment has full control over which gems are installed there and the only piece of information needed for promotion is the manifest file declaring specific versions to promote.


The Reality

We built the tools and dev/qa gem servers and started pushing gems to QA using the described approach. When the time came to implement it in production, our operations team rejected it. They had a reason: they wanted a unified approach for delivering all packages they need to install. Since we have Java and many other non-ruby applications while the ops requirement was to use RPMs for package distribution, we had to adapt. We retained the dev gem sever and continued to allow developers to push gems to it. We continue to generate the manifest file, but instead of delivering it as is, we use it to pick up the specific versions of gems and repackage them as RPMs (using a modified version of gem2rpm). They are then delivered to QA and later to production.


The Future

We still believe that using the 'gem servers' hierarchy is the right way for plugem-based applications distribution. Most likely the original scripts will be a part of the near future plugem public release. We hope that one day it might become the preferred method of rails applications distribution.

Tuesday, March 06, 2007

Gem-based Deployment and Delivery: Part 1 - When Capistrano Is Not Enough

Part of the Gem-based Development Process series

Introduction


In the beginning, we, as many other rails projects, were using Capistrano for deployment. This changed when it came time to start preparing builds for QA and production. Having formal QA and Operations teams, we had to adjust our approach to meet their requirements for deployment. Another force pushing us off Capistrano--even for deployment in the development environment--was the existence of multiple servers where different versions of the application could be deployed by anyone from the large development team. As a result, we built our own deployment tools to be used by QA/Operations (and occasionally the development team.) We also use push-button builds (via luntbuild) to deploy applications to dev.


The Tools

A New RHG Developer's Illustrated Primer provides insight into how deployment tools are used on our projects. Since it stops at the point when the build is ready for QA, it does not show all use cases; however, it gives up enough to see what might be going on later in QA and production. The teams there use rhgcontrol for any application or shared component deployment task. Wrapping up all deployment activities in a self-contained tool gives the development team better control over how applications are installed and deployed in those environments. In addition, it allows the addition of new features to the existing command set without changing the installation instructions.

A couple of examples:

  1. When our DBAs asked us to provide an automated way to update the application history table with the current application upon deployment, we added auto-generation of a db migration that does just that to the rhgcontrol migrate command.
  2. We want to know what versions of shared components are used by the running version of an application. Instead of building a runtime configuration querying system a-la JMX, we simply fixed rhgcontrol start to dump its runtime configuration right after the application was started.

The Deployment

Any application or shared component is potentially deployable. All that is required is a single file--deployment.yml--in the config directory of the gem (since all applications at RHG are plugems). The content of the file is read and executed at deployment time when rhg deploy is run. The deployment configuration is simpler than a Capistrano recipe and contains just a few sections.

Sample deployment.yml:

# For servers
server: &server_cfg

files:
- database.yml
- log4r.xml

execs:
- ln -nfs <%= @app_home %> <%= @app_install_dir %>/<%= @app_name %>
- <%= @cmd_host_gems %>
- rake some:action


qa: *server_cfg
dev: *server_cfg

# For developer's workstations
default:

files:
- database.yml

execs:
- <%= @cmd_link_to_app %>

First, there is a separation by a deployment environment - dev/qa/etc. Those names are based on the hostname of a machine where rhg deploy is executed. We adopted unified DNS naming conventions for our rails boxes (e.g. all of our QA machine names start with qa.) This allows us to define host classes and put, say, QA-specific tasks. If the hostname does not match any class, the default section is used.

There are two types of the deployment tasks: copying over configuration files (the files subsection) and execution of commands (the execs section).

Some configuration files, such as database.yml, are often deployment environment specific. The deployment-time configuration directory (config/deployment) may contain such files, prefixed with either the host class (like dev) or the full hostname (like dev8-rails.) They are used to overwrite the copies under the config/ directory at deployment time.

Configuration-file overrides are a small piece of the multiple-configuration puzzle we've had to solve at RHG. We plan to have a separate post on this subject, which will cover runtime configuration as well.

There are two types of the execution commands--regular UNIX commands and macros. Macros are commands that are bundled with the rhg tool (when it makes sense to share them between different applications.) For example, the macros for hosting gems (@cmd_hosts_gems) looks like this:

- rm -rf gems
- tar -xf <%= @deployment_gem_dir %>/templates/gems-bare.tar

Instead of putting those commands in the deployment.yml file of every application that uses the locking mechanism, we share them via macros. They also provide us a single place to change the implementation of these common tasks.

The execs section leverages deployment-specific variables, resulting in a system flexible enough to handle every deployment task we've encountered.


Conclusion

While Capistrano is suitable for most of the rails projects out there, we had to build our own deployment and delivery tools to wrangle our multi-environment, multi-application, multi-component portal. We plan to extract and release the useful parts of the tool for Rails development teams that are isolated from their QA/Production environments.

Friday, March 02, 2007

Data Loaders in Migrations

Two cases of data usage at RHG (as outlined in the DB Migrations With a Twist post) - migration consolidations and test data creation via migrations, led us to having migrations that just load and execute SQL statements from a file. Another use case is mass-loading of referential data. To help with those tasks we use a simple module that provides a method for data loading:

module DataLoader

def load_data(name = 'data', ext = 'sql')
file_name = data_file(name, ext) || fail("Cannot find the data file for #{ name }")
execute_sql_from_file(file_name)
end

def execute_sql_from_file(file_name)
say_with_time("Executing SQL from #{ file_name }") do
IO.readlines(file_name).join.gsub("\r\n", "\n").split(";\n").each do |s|
execute(s) unless s == "\n"
end
end
end

private

def data_file(name, ext)
mode = ENV['RAILS_ENV'] || 'development'
[
data_file_name(mode + '_', name, ext),
data_file_name('', name, ext)
].detect { |file_name| File.exists?(file_name) }
end

def data_file_name(prefix, name, ext)
"#{ RAILS_ROOT }/db/migrate/data/#{ prefix }#{ name }.#{ ext }"
end

end

A sample migration using the method for loading the consolidated data to all DBs looks like this:

class ConsolidatedMigrations < ActiveRecord::Migration

extend DataLoader

def self.up
all_dbs.each_key do |name|
run_on_db(name) { load_data "consolidated_for_#{ name }" }
end
end

end

It loads Rail-environment specific files from the db/migrate/data/ directory:

development_consolidated_for_main.sql
development_consolidated_for_portal_referential.sql
...
production_consolidated_for_main.sql
production_consolidated_for_portal_referential.sql
...
test_consolidated_for_main.sql
test_consolidated_for_portal_referential.sql