[PLUGIN RELEASE] ActsAsOverflowable

acts_as_overflowable

Written by Nicholas Lega

== DESCRIPTION:

Allows a column to overflow data into a secondary column if the data size exceeds the character limit. This is useful for fast indexing.

Instead of trying to index text blobs, you can specify a varchar column to be used for indexing. Any characters that do not fit into the base column will be automatically saved into or retrieved from an overflow field.


== SYNOPSIS:

Create overflow column migration:

 class CreateOverflowModels < ActiveRecord::Migration   
   def self.up         
     create_table :overflows do |t|       
       t.integer :overflowable_id       
       t.string :overflowable_type       
       t.string :overflow       
       t.timestamps     
     end

     add_column(:my_class_that_uses_overflows,
                :has_overflow,                 
                :boolean)     

     add_column(:my_class_that_uses_overflows,                 
                :overflowable_text,                 
                :string,                 
                :limit => 255)
   end

   def self.down
     drop_table :overflows

     remove_column(:my_class_that_uses_overflows,
                   :has_overflow)

     remove_column(:my_class_that_uses_overflows, 
                   :overflowable_text)
   end
 end

Add to model:

 class MyClassThatUsesOverflow < ActiveRecord::Base   
   acts_as_overflowable :overflow_column => :overflowable_text,
                        :overflow_limit => 255,
                        :overflow_indicator => :has_overflow,
                        :overflow_table => :overflows
 end

To get and set the value of the "overflowable_text" field as described in the test model, use:

 # #{my_field_name_here}_with_overflow is the method name
 long_text = overflowable_obj.overflowable_text_with_overflow

 overflowable_obj.overflowable_text_with_overflow = very_long_text

Simply using the getters and setters for "overflowable_text" without appending "_with_overflow" to the method name will return the fragment of the text contained in the base column.


== INSTALL:

sudo gem install revolutionhealth-acts_as_overflowable -s http://gems.github.com


== SOURCE:

http://github.com/revolutionhealth/acts_as_overflowable/tree/master


== FEATURES/PROBLEMS:

* Only tested on mysql and sqlite3

[PLUGIN RELEASE] ActsAsSeoFriendly

ActsAsSeoFriendly

== DESCRIPTION:

Create an SEO friendly field for a model automatically based on a given field.

So if you have a Blogs model, and you would like create an SEO friendly version
of the 'title' field, you would just add this to your model and then be able to
use the SEO friendly id as the unique id to the resource. The plugin will only
append an integer to the SEO id if there is a collision.


== SYNOPSIS:

Create seo column migration:

class CreateSeoTestModels < ActiveRecord::Migration
  def self.up
    create_table :seo_test_models do |t|
      t.string :name
      t.timestamps
    end
  SeoTestModel.create_seo_friendly_column()
  end

  def self.down
  SeoTestModel.drop_seo_friendly_column()
    drop_table :seo_test_models
  end
end

Add to model:

 class SeoTestModel < ActiveRecord::Base
  acts_as_seo_friendly :resource_id => :name,
              :seo_friendly_id_field => :seo_id, # default is :seo_friendly_id
              :seo_friendly_id_limit => 100 # default is 50
 end

To lookup the resource in the controllers use:

 SeoTestModel.find_by_seo_id(params[:id])

== INSTALL:

sudo gem install revolutionhealth-acts_as_seo_friendly -s http://gems.github.com

== SOURCE:

http://github.com/revolutionhealth/acts_as_seo_friendly/tree/master

To see the plugin in action see our recipes section, for example:

http://www.revolutionhealth.com/recipes/thai-miang-khem-style-salad


== FEATURES/PROBLEMS:

* Only tested on mysql and sqlite3

Acts As Fast But Very Inaccurate Counter

Introduction

If you have chosen the InnoDB MySQL engine over MyISAM for its support of transactions, foreign keys and other niceties, you might be aware of its limitations, like much slower count(*). Our DBAs are in a constant lookout for slow queries in production and the ways to keep DBs happy so they recommended that we should try to fix count(). They suggested to check SHOW TABLE STATUS for an approximate count of rows in a table. This morning I wrote acts_as_fast_counter which proved that the speed is indeed improved but the accuracy might be not acceptable. The rest of the post just records details of the exercise.

The approach

I created a model per engine and seeded each with 100K records. Then I run count on each model for a thousand times and measured the results.

The code:

module ActiveRecord; module Acts; end; end 

module ActiveRecord::Acts::ActsAsFastCounter
  
  def self.included(base)
    base.extend(ClassMethods)
  end
  
  module ClassMethods
    
    def acts_as_fast_counter
      self.extend(FastCounterOverrides)
    end
    
    module FastCounterOverrides
      
      def count(*args)
        if args.empty?
          connection.select_one("SHOW TABLE STATUS LIKE '#{ table_name }'")['Rows'].to_i
        else
          super(*args)
        end
      end

    end
    
  end
  
end

ActiveRecord::Base.send(:include, ActiveRecord::Acts::ActsAsFastCounter)

# create_table :myisams, :options => 'engine=MyISAM'  do |t|
#   t.column :name, :string
# end
# 100_000.times { Myisam.create(:name => Time.now.to_s) }
# 
# create_table :innodbs, :options => 'engine=InnoDB' do |t|
#   t.column :name, :string
# end
# 100_000.times { Innodb.create(:name => Time.now.to_s) }

class Bench
  
  require 'benchmark'
  require 'acts_as_fast_counter'

  def self.run
    measure
    show_count
    convert_to_fast_counter
    show_count
    add_records
    show_count
    destroy_records
    show_count
    measure
  end

  def self.measure
    puts "* Benchhmarks:"
    n = 1_000
    Benchmark.bm(12) do |x|
      x.report('MyISAM') { n.times { Myisam.count } }
      x.report('InnoDB') { n.times { Innodb.count } }
    end
  end
  
  def self.convert_to_fast_counter
    Innodb.send(:acts_as_fast_counter)
    puts "* Converted Innodb to fast counter"
  end
  
  def self.add_records
    @myisam = Myisam.create(:name => 'One more')
    @innodb = Innodb.create(:name => 'One more')
    puts "* Added records"
  end
    
  def self.destroy_records
    @myisam.destroy
    @innodb.destroy
    puts "* Destroyed records"
  end
  
  def self.show_count
    puts "* Record count:"
    puts "    MyISAM: #{ Myisam.count }"
    puts "    InnoDB: #{ Innodb.count }"
  end
  
end

The results:

* Benchhmarks:
                  user     system      total        real
MyISAM        0.180000   0.040000   0.220000 (  0.289983)
InnoDB        0.430000   0.070000   0.500000 ( 35.102496)
* Record count:
    MyISAM: 100000
    InnoDB: 100000
* Converted Innodb to fast counter
* Record count:
    MyISAM: 100000
    InnoDB: 100345
* Added records
* Record count:
    MyISAM: 100001
    InnoDB: 100345
* Destroyed records
* Record count:
    MyISAM: 100000
    InnoDB: 100345
* Benchhmarks:
                  user     system      total        real
MyISAM        0.250000   0.030000   0.280000 (  0.350673)
InnoDB        0.250000   0.040000   0.290000 (  0.977711)

Final thoughts

The MySQL manual has a clear warning about inaccuracy of the amount of rows in the SHOW TABLE STATUS results:

Rows - The number of rows. Some storage engines, such as MyISAM, store the exact count. For other storage engines, such as InnoDB, this value is an approximation, and may vary from the actual value by as much as 40 to 50%. In such cases, use SELECT COUNT(*) to obtain an accurate count.

The test confirms it by showing 345 more records then expected thus making it not very useful but for some edge cases. If you know a way to improve the speed of count() on InnoDB with some other approach beyond using a counter table, please share.

Plugin I Cannot Live Without

The Enhanced Rails Migrations plugin was written to end the constant battle we had with clashing names in db migrations within our large development team. We tried everything: special commit policies, rake tasks, even claiming the next migration number in subversion. Nothing worked and CI server was sending 'broken build due to conflicting migration number' messages almost daily. Since the plugin was introduced to all our rails applications around six months ago, I have not heard of a single case of conflicting migrations. Seemingly, the goal was well accomplished.

What I found over time, is that the plugin is not only useful for large projects. Any rails development effort with more than one programmer involved benefits from using it. If you ever had to renumber your new migration after doing svn up you know what I am talking about. It makes sense to install this plugin as the very first one in your project since an amount of migrations at the beginning tends to grow much faster then later in the game.

The plugin works for rails versions 1.1.6 up to the latest edge. When you start your next project with multiple developers, use it and you should be able to forget that you ever had problems with clashing migrations.

Moving models to a different database

There many reasons to use multiple databases (DBs) and when this is done, there is often a case when a model needs to be moved from one DB to another. The impetus could be that part of the data is referential and this is being reflected by moving it to a read-only DB. Another possibility is that we want to protect some data with an additional layer of security, so we extract it to a secure DB. In all cases, the challenge is to migrate the existing data. When the amount of data is considerably large, there is no choice but to do it via SQL data loaders or similar techniques. On the other hand, if it is acceptable to leverage rails db migrations and you prefer to do any data manipulation through them, there are some challenges to face.

Often, you need to have an access to old and new models during data migration. One solution is to move or copy an existing model to a separate namespace and put the new model instead at the old namespace. Let's look at a couple of examples:


Extracting referential data

I have a model Fruit in our main DB which gets its data from an external source, so we only access it read-only. We want to enforce it by moving the data to a DB which we access with a read-only account. First, I create a referential_db entry in database.yml:

dbs:
  database: main

  referential_db:
    database: referential

Then, I copy the original model, Fruit, to a dedicated namespace, so the model becomes RetiredModels::Fruit. I add establish_connection to the original namespace model:

# create_table :fruits do |t|
#   t.column :name, :string
# end
class Fruit < ActiveRecord::Base
  establish_connection configurations[RAILS_ENV]['referential_db']  
end

Everything is set for migration. Since it is a referential data, the migration needs to preserve data integrity so the models belonging to Fruit can still reference it by an old id:

def self.up
  RetiredModels::Fruit.find(:all).each do |old_record|
    Fruit.new(old_record.attributes) { |new_record| new_record.id = old_record.id }.save!
  end
end

After the successful migration run, all data is replicated to a new DB. The retired model can be removed during next deployment, and the original table dropped.

There is one caveat for development and test modes. If you don't want to bother with multiple databases in those modes, you need to take care of having no table name clashing. So, the new model would have to use different table names via set_table_name.


Securing sensitive data

One of the models belonging to Fruit is SecretFruit. It contains a secret name for every fruit out there. Our legal department asked the development team to protect that data in case our DB is stolen. We decided to migrate the existing SecretFruit data to a protected DB and keep sensitive data encrypted with help from Acts As Secure. First, I create a secure_db entry in database.yml:

dbs:
  database: main

  secure_db:
    database: secure
    host: protected_host

Then, I copy the original model, SecretFruit, to a dedicated namespace, so the model becomes RetiredModels::SecretFruit. I modify the model in the original namespace to reflect the new requirements:

# create_table :secret_fruits do |t|
#   t.column :name, :binary
#   t.column :fruit_id, :integer
# end
class SecretFruit < ActiveRecord::Base
  establish_connection configurations[RAILS_ENV]['secure_db']
  acts_as_secure :crypto_provider => MasterKeyProvider
  belongs_to :fruit
end

Since data encryption is done on-the-fly and there are no data integrity requirements, the migration is straightforward:

def self.up
  RetiredModels::SecretFruit.find(:all).each { |old| SecretFruit.create!(old.attributes) }
end

I can now safely delete RetiredModels::SecretFruit and associated data.

DRYing Models via Acts As

ActsAs is an idiom familiar to every Rails developer, which makes it a good candidate for a shared functionality between models. Using it as early in the game as possible allows one to work on its functionality without a need to touch the code in multiple models. Let's look at a couple of examples.

Acts As Unique

I have some models that I want to have uniqueness across my application. I use some UUID mechanism (initially, a db call) to set a field (:token) after creation. Since I have multiple models, I decide to extract it the code for uniqueness setting to acts_as_unique. After refactoring, my model Fruit looks like:

# create_table :fruits do |t|
#   t.column :name, :string
#   t.column :token, :string
# end
class Fruit < ActiveRecord::Base
  acts_as_unique
end

My acts_as_unique might look like:

module ActiveRecord; module Acts; end; end
module ActiveRecord::Acts::ActsAsUnique
  
  def self.included(base)
    base.extend(ClassMethods)
  end
  
  module ClassMethods
    def acts_as_unique(field = :token)
      validates_uniqueness_of field 
      before_validation_on_create do |o|
        o.send("#{ field }=", connection.select_one('SELECT UUID() AS UUID', "#{name} UUID generated")['UUID'])
      end
    end    
  end
end

ActiveRecord::Base.send(:include, ActiveRecord::Acts::ActsAsUnique)

Let's try it:
>> f = Fruit.create(:name => 'apple')
>> p f.token
"0a4d7c46-4df0-102a-a4b9-59b995bffdb7"

Now I can work on acts_as_unique to replace the DB call with a UUID gem or some other implementation without affecting the rest of the code.


Acts As Trackable

I have some models for which I want to keep track of when instances are created or updated. I have a polymorphic Event model for storage of such events. Since there are multiple models I want to track, I extract the functionality to acts_as_trackable. After refactoring, my models look like:

# create_table :fruits do |t|
#   t.column :name, :string
# end
class Fruit < ActiveRecord::Base
  acts_as_trackable
end

# create_table :events do |t|
#   t.column "action",              :string
#   t.column "created_at",          :datetime, :null => false
#   t.column "trackable_type",      :string
#   t.column "trackable_id",        :integer
# end
class Event < ActiveRecord::Base 
  belongs_to :trackable, :polymorphic => true
end

module ActiveRecord; module Acts; end; end 
module ActiveRecord::Acts::ActsTrackable
  
  def self.included(base)
    base.extend(ClassMethods)
  end
  
  module ClassMethods
    def acts_as_trackable
      has_many :events, :as => :trackable, :dependent => :destroy
      after_update { |o| o.events.create(:action => 'updated') }
      after_create { |o| o.events.create(:action => 'created') }
    end
  end
  
end

ActiveRecord::Base.send(:include, ActiveRecord::Acts::ActsTrackable)

Let's see what we got:
>> f = Fruit.create(:name => 'apple')
>> p f.events.collect(&:action)
["created"]
>> f.name = 'passionfruit'
>> f.save!
>> p f.events.collect(&:action)
["created", "updated"]

The Event model is likely to evolve but it would be easier to support it since the only place where I need to reflect the changes is acts_as_trackable. The goal is achieved.

[PLUGIN RELEASE] ActsAsSecure

Introduction

ActsAsSecure adds an ability to store ActiveRecord model's fields encrypted in a DB. When a model is marked with acts_as_secure, the :binary type fields are recognized as needed to be stored encrypted. The plugin does before_save/after_save/after_find encryption/decryption thus making it transparent for a code using secured models.

The plugin supports a master key approach as well as individual records encryption keys. It does not contain any crypto provider but allows to plug in any external one as long as it supports encrypt/decrypt methods.

The fields are converted to a YAML form before encryption. After description they are restored via YAML.load. Since fields are stored encrypted, the find usage is very limited.

Usage

Master Key Provider Usage

class SecureModel < ActiveRecord::Base
  acts_as_secure :crypto_provider => MasterKeyProviderClassOrInstance
end

SecureModel.create()
SecureModel.find(:first)

Individual Keys Provider Usage

class SecureModel < ActiveRecord::Base
  acts_as_secure
end

SecureModel.with_crypto_provider(SomeProvider.new(some_param)) { SecureModel.find(:first) }
SecureModel.with_crypto_provider(SomeProvider.new(some_param)) { SecureModel.create() }

Other Options

acts_as_secure :storage_type => :text -- changes the secure storage type from :binary to the supplied one
acts_as_secure :except => [:field1, :field2] -- disables the secure behavior for the :binary type fields in a supplied list


Example

Let's define three models: Fruit (not secure), SecretFruit (uses the master key approach), and UberSecretFruit (uses the individual key approach).

Rot13CryptoProvider represents a master key provider. SaltedRot13CryptoProvider depends on a salt individual for each record.

Models

class Fruit < ActiveRecord::Base
  has_one :secret_fruit
  has_one :uber_secret_fruit 
end

class CreateFruits < ActiveRecord::Migration
  def self.up
    create_table :fruits do |t|
      t.column :name, :string
    end
  end
end

class Rot13CryptoProvider
  class << self
    def encrypt(arg)
      arg.tr("A-Za-z", "N-ZA-Mn-za-m")
    end
    alias_method :decrypt, :encrypt
  end
end

class SecretFruit < ActiveRecord::Base
  acts_as_secure :crypto_provider => Rot13CryptoProvider
  belongs_to :fruit
end

class CreateSecretFruits < ActiveRecord::Migration
  def self.up
    create_table :secret_fruits do |t|
      t.column :name, :binary
      t.column :fruit_id, :integer
    end
  end
end

class SaltedRot13CryptoProvider
  def initialize(salt)
    @salt = salt
  end
  def encrypt(arg)
    @salt + arg.tr("A-Za-z", "N-ZA-Mn-za-m")
  end  
  def decrypt(arg)
    arg[@salt.size .. -1].tr("A-Za-z", "N-ZA-Mn-za-m")
  end  
end

class UberSecretFruit < ActiveRecord::Base
  acts_as_secure
  belongs_to :fruit 
end

class CreateUberSecretFruits < ActiveRecord::Migration
  def self.up
    create_table :uber_secret_fruits do |t|
      t.column :name, :binary
      t.column :fruit_id, :integer
    end
  end
end

Usage

>> f = Fruit.create(:name => 'passion fruit')
>> SecretFruit.create(:name => 'maracuya', :fruit => f)
>> puts f.secret_fruit.name
maracuya
>> secret = readline.chomp
uber_secret
>> crypto_provider = SaltedRot13CryptoProvider.new(secret)
>> UberSecretFruit.with_crypto_provider(crypto_provider) { UberSecretFruit.create(:name => 'Passiflora edulis', :fruit => f) }
>> UberSecretFruit.with_crypto_provider(crypto_provider) { puts f.uber_secret_fruit.name }
Passiflora edulis

DB

> select * from secret_fruits;
+----+---------------+----------+
| id | name          | fruit_id |
+----+---------------+----------+
|  1 | --- znenphln  |        1 | 
+----+---------------+----------+

> select * from uber_secret_fruits;
+----+-----------------------------------+----------+
| id | name                              | fruit_id |
+----+-----------------------------------+----------+
|  1 | uber_secret--- Cnffvsyben rqhyvf  |        1 | 
+----+-----------------------------------+----------+

Installation

As plugin:
script/plugin install svn://rubyforge.org/var/svn/acts-as-secure/trunk/vendor/plugins/acts_as_secure


License

ActsAsSecure released under the MIT license.


Support

The plugin RubyForge page is http://rubyforge.org/projects/acts-as-secure