Data Loaders in Migrations

Two cases of data usage at RHG (as outlined in the DB Migrations With a Twist post) - migration consolidations and test data creation via migrations, led us to having migrations that just load and execute SQL statements from a file. Another use case is mass-loading of referential data. To help with those tasks we use a simple module that provides a method for data loading:

module DataLoader

  def load_data(name = 'data', ext = 'sql')
    file_name = data_file(name, ext) || fail("Cannot find the data file for #{ name }")
    execute_sql_from_file(file_name)
  end

  def execute_sql_from_file(file_name)
    say_with_time("Executing SQL from #{ file_name }") do
      IO.readlines(file_name).join.gsub("\r\n", "\n").split(";\n").each do |s|
        execute(s) unless s == "\n"
      end
    end
  end

private
    
  def data_file(name, ext)
    mode = ENV['RAILS_ENV'] || 'development'
    [ 
      data_file_name(mode + '_', name, ext),
      data_file_name('', name, ext)
    ].detect { |file_name| File.exists?(file_name) }
  end 

  def data_file_name(prefix, name, ext)
    "#{ RAILS_ROOT }/db/migrate/data/#{ prefix }#{ name }.#{ ext }"
  end
  
end

A sample migration using the method for loading the consolidated data to all DBs looks like this:

class ConsolidatedMigrations < ActiveRecord::Migration

  extend DataLoader

  def self.up
    all_dbs.each_key do |name|
      run_on_db(name) { load_data "consolidated_for_#{ name }" }
    end
  end

end

It loads Rail-environment specific files from the db/migrate/data/ directory:

development_consolidated_for_main.sql
development_consolidated_for_portal_referential.sql
...
production_consolidated_for_main.sql
production_consolidated_for_portal_referential.sql
...
test_consolidated_for_main.sql
test_consolidated_for_portal_referential.sql

[PLUGIN RELEASE] Enhanced Rails Migrations v1.1.0

Enhanced Rails Migrations is a plugin that makes easier rails development in a large team with multiple code branches.

Native rails db migrations are great for most of the teams. Troubles begin when many developers try to add a migration or when code is merged between branches. Since native migrations are based on incremental numbers, the name clashing is frequent. Same happens with a merged code on even larger scale. This plugin was born at the Revolution Health Group team to address the problems with large multi-branched development (see our blog entry - revolutiononrails.blogspot.com/2007/01/db-migrations-usage-at-rhg.html) where it has been successfully used for a few last months.

The Solution

There is a couple of patches against the Edge rails approaching the problem from different angles. You might want to try them unless you work on 1.1.6 or prefer plugins to handle such things.

The plugin monkey-patches ActiveRecord classes to replace the sequential number based rails migration mechanism with the timestamp based one. That allows to avoid name collisions. In addition, it maintains a tracking table of already run migrations instead of the standard single number schema_info. The table is being used for decision whether to run a migration. The biggest difference from the standard way is that migrations below the current schema version are still applied if they have not been applied yet (not in the tracking table).

Important: See the First Run section for details how to use it in existing projects.

Compatibility

The code has been used mostly on 1.1.6 and it was tested against the edge rails.

Setup

Installation

Install as any other Rails plugin:
ruby script/plugin install svn://rubyforge.org/var/svn/enhanced-mgrtns/enhanced_migrations

Usage

There is no difference in migration usage - use script/generate migration to generate a new migration and rake db:migrate for migration.

schema_version is depreciated. If you need to find the last run migration, use: SELECT MAX(id) FROM migrations_info

First Run

If you are adding the plugin in a middle of the development process, when some migrations have been created and applied, you need to mark those as already run or rake db:migrate would try to apply them again. The plugin contains a sample zero-number migration which queries the DB for the last run migration number and marks all migrations prior to the number as already run in the new tracking table. It also deletes the schema_version table. To use it, copy migrations/000_run_migrations_marker.rb from the project to your rails db/migrate directory.

License

Enhanced Rails Migrations is released under the MIT license.

Support

The plugin RubyForge page is rubyforge.org/projects/enhanced-mgrtns

Plugems Dependency Loading: Part 2 - Bundling a plugin as a Gem and getting it all to work

Plugins live in the project root, so Rails need only loop through the plugins, unshift them onto the load path, and require their init files. In a plugem environment, it's a little more complicated (to implement, not to use.)

Each of our applications has a manifest file. This is simply a YAML file that has some basic metadata about the appication (name, version, etc.) as well as a list of all of that application's plugem dependencies. This manifest file has two applications:

1. Our rake tasks use it to generate a gemspec file for application gemification
2. The bootstrapper uses it as a list of dependent plugems to load
   

Without getting into the weeds, this is accomplished in the following way:

1. A bootstrap set of tasks is loaded right after the environment is booted.
2. The bootstrap monkeypatches the Rails initializer, alias-chaining our plugem-loading code to plugin load.
3. Upon startup, the plugem bootstrap code will loop through the dependencies and load and initialize them all. This is all mostly handled via require_gem, as our gemspec generator automatically registers the equivalent 'init.rb' type initialization via gem autorequire.

Next time: We put views, rake tasks, and layouts in these things too!

InfoQ

Brian Ketelson of InfoQ was gracious enough to feature Revolution Health in an article today. Thanks Brian.

http://www.infoq.com/news/2007/02/revolution-health-profile

Here's a link to all of their Ruby-oriented articles:

http://www.infoq.com/ruby/

Locking down a deployed application

A single host might have multiple applications installed each on a different release track. Since all our applications are distributed as a bunch of gems, there is a potential issue with shared components. Even though the whole portal is being tested when some application or component is being pushed through the stack, there is always a possibility of incompatibility of on of the upgraded shared component with another dependent application. In some other environments there would only one solution - to roll back the whole new application. In our environment we have another option - to lock down the affected application. When an application is deployed, one of the deployment steps is to create a gem repository inside the application structure. The application gem repository has the sources gem installed and the rest of gems sym-linked as in this example:

actionmailer-1.2.5 -> /usr/lib/ruby/gems/1.8/gems/actionmailer-1.2.5
actionpack-1.12.5 -> /usr/lib/ruby/gems/1.8/gems/actionpack-1.12.5
...
sources-0.0.1

Gems put into an internal gem structure are those it was tested with, so they are guarantee to work. All is needed to lock down an application now is to set the GEM_HOME environment variable on a web server instance, serving the application, and to point it to the application internal gem repository. After that the application won't use the latest versions of the gems but those specific it is locked to.