Gem-based Deployment and Delivery: Part 2 - Distribution via Gem-servers

Part of the Gem-based Development Process series:

The Idea

Once we packaged all of our applications and shared components as gems, it did not take us long to realize that we could leverage the native method of gem distribution - gem servers - to build an infrastructure for pushing products through the dev/qa/production deployment stack. The idea of gem promotion was born.

There are three layers of gem servers - one per deployment environment (dev/qa/production). Gems can be promoted from the upstream gem server to the downstream one. The direction of promotion is dev->qa->prod. The individual boxes within the deployment environment install gems only from the server that serves their environment. For example, when a QA person installs gems on qa7-rails & qa8-rails boxes, he uses the qa-gems server as the source (as in gem install xyz --source qa-gems --remote). The dev gem server is the base one where freshly built gems are pushed to.


The Implementation

Since our QA and Operations teams wanted to have full control over what they have on their gem servers, we adopted the push-pull model instead of a more simple push-only one (when a gem promotion works by pushing from the upstream to the downstream). In our current model, developers push gems to the base (dev) gem server first. After it is tested and is QA ready, the manifest file, containing a specific gem name and version (together with names/version of all dependent gems), is generated and sent to QA. A QA installer uses the manifest to pull gems from the upstream dev gem server to the downstream QA one. He then goes to the individual QA boxes and upgrades the gems there. When QA clears the build, the manifest is sent to Operations to repeat the procedure in production with the qa gem server being the upstream one.

We greatly simplified the usage by hiding decisions about upstream/downstream, the source gem, etc. behind a single tool that was deployment environment aware. For example, to pull down gems based on the manifest to the qa gem server, the installer issues a command rhg pull manifest_file on the qa gem server box. The rhg tool queries the hostname of the machine it is being run on, and based on its name (qaX-YYY), it picks the upstream server (dev-gem). Running rhg up gem_name gets the gem (and all of its dependencies) from the environment specific gem server, with the base (dev) gem server being the default option so developers can use the same command for updating their local gem repositories.

Under this scheme, each environment has full control over which gems are installed there and the only piece of information needed for promotion is the manifest file declaring specific versions to promote.


The Reality

We built the tools and dev/qa gem servers and started pushing gems to QA using the described approach. When the time came to implement it in production, our operations team rejected it. They had a reason: they wanted a unified approach for delivering all packages they need to install. Since we have Java and many other non-ruby applications while the ops requirement was to use RPMs for package distribution, we had to adapt. We retained the dev gem sever and continued to allow developers to push gems to it. We continue to generate the manifest file, but instead of delivering it as is, we use it to pick up the specific versions of gems and repackage them as RPMs (using a modified version of gem2rpm). They are then delivered to QA and later to production.


The Future

We still believe that using the 'gem servers' hierarchy is the right way for plugem-based applications distribution. Most likely the original scripts will be a part of the near future plugem public release. We hope that one day it might become the preferred method of rails applications distribution.