Tuesday, May 01, 2007

[RELEASE] Plugems Runtime

One sentence summary: "Everything is a gem, most things work from within the gem, and all dependencies are accounted for."

How are Plugems different from...

Plugins

  1. Plugems are a superset of Rails plugins.
  2. Plugems are bundled as Rubygems and are installed on the server hosting your application. Plugins are directories of files that get copied into the vendor/plugins directory of every application that leverages them.
  3. Plugins aren't dependency-managed. If one plugin depends upon another, then you have to handle the dependencies in a very implicit way. Plugems supports full dependency management: applications declare dependencies, dependencies declare their own subdependencies. Plugin load-path hacks aren't enough because they fail to address the issue of 'behavioral dependencies' (i.e. one plugin's init needing run after another.)
  4. Thanks to Rubygems, Plugems supports revision/release conventions. It was important for us to version dependencies explicitly or implicitly. Now you have full control.
  5. Plugems can be distributed through the standard Rubyforge repository (but they're still backwards compatible with plugin install.)
  6. Plugems adds support for the loading of views from gems (in addition to rake tasks, models, etc.)
  7. With plugins, you can't update a plugin for all apps on a box without pushing the applications. With Plugems, you simply install the new gem and bounce the apps. All applications that are not hard-locked to a particular revision of that plugem will pick up the latest version automatically.
Note: We still recommend plugins for quick and dirty things that you don't expect to undergo a heavy rate of change or become a shared dependency. At RHG, most Plugems start their life as plugins (and get promoted once multiple applications want to share it.)

Engines

  1. Engines have no support for dependency management or revision/release conventions (see #2, #3 for plugins.)
  2. With Engines, you can't update an engine for all apps on a box without pushing new application code. (see plugins #7)
  3. The Plugems library is distributed as a ruby gem -- no plugin or svn external to maintain. Simply update your gem when you're ready to take a new version.
  4. Plugems intentionally don't support migrations -- It was a fundamental choice we made early on to avoid the temptation to build apps that share databases (resulting in highly-coupled applications with the database as the integration point.)
  5. Plugems intentionally don't support runtime asset loading. We believe the coalescence of assets is a deployment-time issue (plugem deployment is forthcoming in this series.) For local development, something like this is needed, though. We use a mongrel handler for local development.
  6. Plugems intentionally don't support to routes. We decided that applications own urls, not shared libraries.


Recent related core discussion
  1. This patch does not account for rake tasks, views, etc. from within plugin gems (albeit probably intentionally for consistency with plugins.)
  2. With the latest patch, gems are bundled with the application, preventing live-uptake of new versions (see Engines #5)

Other notable features

  1. Plugems is accompanied by a tool for examining, updating, building, and 'deploying' plugems to other developers on the team (or developers on other teams that share your plugem.). Think `svn up; plugem up`. These capabilities leverage the existing Rubygems platform. We're still busy preparing these for release--this release is only the runtime environment.
  2. Support for applications-as-gems (more useful with the release of the deployment-time tools.)
  3. You can develop a plugem without loosing its dependency knowledge. You can check out or symlink the plugem into your plugins directory for active development and Plugems will ignore the installed gem but still load that 'bypassed' Plugem's dependencies.

Installation Guide:

Install it on your box:

As a gem:
gem install plugems

Create your plugem configuration:

Add config/manifest.yml describing your plugem and its dependencies:
:version: [1, 0]
:name: "cool_application"
:description: "My First Plugemified Application"
:dependencies:
- ['some_gem', '~> 1.0']
- ['other_gem', '> 2.0']
- ['one_more', '2.0.1']

Bootstrap the plugems:

Add the plugems requirement to config/boot.rb at the bottom right before the initializer call:
  # Add this line:
require_gem 'plugems', '~> 1.0'

Rails::Initializer.run(:set_load_path)
end


You are all set to enjoy the power of plugems!

License

Plugems released under the MIT license.


Support

The plugin RubyForge page is http://rubyforge.org/projects/plugems

We've been using this internally for quite a few months, and are really interested in feedback from the community. Please tell us what you think!

3 comments:

James Adam said...

Hi guys!

This looks nice, but I think you've perhaps misrepresented a few aspects. I'll go through 'em here, and we can see what you think.

As a preface to the whole thing, there's no such thing as an "engine" anymore ;) So, as you indicate, points 1, 2 and 3 under the engines section apply equally to plugins and "engines", because there is no underlying difference between them. I'm not sure that it makes sense to highlight them again, but then as I see it, the "big win" that Plugems gives is dependency management, so I understand why you want to emphasize this.

You mention "updating a shared component on a single machine which then affects all applications"; this, here, is where you need to be super careful:

Firstly, if you vendor everything, the point is moot since your application uses it's vendor version, rather than anything from a shared environment, so updating the Plugem on the server machine won't do anything for these applications.

If you don't vendor everything, then you really need to be sure that your application is requiring a specific version of a Plugem, otherwise your machine-wide update might cause issues with some of your applications. This is a lesson hard-learned by many Rails developers in the past.

So, if you're not vendoring everything, we've learned that you must require specific versions of gems (and this applies equally to SVN externals; Piston is interesting in this respect too) in your application's environment.rb. If you're following this next-to-best-practice, then yup, you'll need to test and deploy a new version to get your application to declare it's dependencies the newly updated Plugems.

What I'm getting at here is that I'm not sure that "machine-wide updates" is the best selling point for a tool that enhances development practices; it may end up causing more headaches than it solves. Caveat emptor!

Regarding plugin migrations - I think you might have the wrong end of the stick. By providing a common migration for a particular plugin feature, you're not producing an integration database. To end up with an integration DB, you'd need to somehow set your database.yml to point to the same machine.

Plugin migrations just make it easy to bundle the schema specification for commonly-used aspects of code (your typical base account system, for example). The migration is run in your database, and you're then free to add columns or modify the table in any way required. It really has nothing to do with integration databases.

Regarding asset loading, I'm not really sure what you mean. Do you propose a secondary mechanism to push a shared javascript library (for example) into deployed applications? It does feel somewhat wrong to put web assets into a gem, but then that's one of the reasons why plugins are nice; they are Rails-specific, so can embrace the concerns of a web application rather than a generic library of code.

And finally, I'm also slightly confused by your views on loading routes, vs. the fact that Plugems support loading of views. The stance the engines plugin takes is that a plugin can provide a default set of routes for its controllers, which the developer can easily include into their route set if they wish. In this manner, the application very much "owns" the set of routes that are actually used.

If Plugems can contain views, can they also contain controllers? If not, how are these views matched to existing controller actions in a way that makes sense across several applications with their own controller structure? Let me know! :)

I know I said "and finally" a few paragraphs back, but if you'll indulge me one more aside, I'm also curious as to how a complex dependency tree fits into the "vendor everything" manifesto. Since one of the most useful aspects of dependencies is only having to state them, rather than ensure that they are explicitly exist in the application, do you intend on freezing all of the dependent Plugems automatically, at the point of deployment?

I'm looking forward to hearing more about your philosophy and implementation :)

Aaron Batalion said...

James,

Thank you for taking the time to comment. We realize our solution isnt perfect, and we really appreciate your feedback and the feedback of the rails community.

Our goal is to help make the community stronger, by sharing how we've dealt with some of the issues you raise.

We've are in the process of posting a few followups to thoughts as blog posts (rather than reply comments).

S. Potter said...

I haven't looked in great detail of how plugems works internally, but the concept of Plugems as opposed to vendor-ing everything is much cleaner.

I must say I find it interesting that some people in the Rails community are still advocating vendor-ing everything even for *controlled* shared environments.

I suspect it might be due to these developers not having experience with professional Rails environments that aren't deployed to anything other than a shared [hosting] environment. I would agree that vendor-ing all your Rails app dependencies for smaller apps that are only designed to run on shared hosts is the only *sane* solution to go with.

However, more and more Ruby developers are working in an enterprise environment, where vendor-ing all dependencies is just a waste of space, time and effort. This also doesn't tackle more complex dependency management scenarios that vendor-ing everything doesn't really solve, it simply hacks (even with piston, which I love for what it is, but it is far for the real enterprise solution out there)!

Those in the uncontrolled shared hosting environments should continue to use vendor snapshots (preferably with piston), as there really is no other sane solution, but it sounds like plugems is attempting to solve a very different problem that engines even when coupled with piston could never solve.

Of course, if your apps don't require this level of abstraction, then it is a moot point and you shouldn't bother if you can't see the utility for your circumstances.

Terminology clarification:
An uncontrolled shared environment might be a shared host (like Site5, Dreamhost, BlueHost, etc.), whereas a controlled shared environment would be a dedicated server that is managed by the same team or IT organization and has multiple Rails applications running.