Managing a Gem Repository with Stickler (2008-10-09)

One of the items Fernand and I mentioned in our Lone Star talk was Stickler. This was a project I had originally started on back in June and it lay dormant for a little while.

The first public version (0.1.0) is now released.

Why should I use Stickler?

One of the problems that we have had at work is inadvertently upgrading a gem on a production system. For example, you are adding a gem to a system that depends on activerecord, >= 2.0.2. This is great, until active_record 2.1.1 comes out and on your production system you type gem install somegem and it updates your active record installation, as a side effect. Yes, you can just quickly uninstall it, but it is annoying.

Another issue that some installations have, is production boxes are unable to talk directly to the Internet. This is a problem when doing a gem installation using rubygems. You either have to run your own gem server, or install the gems locally.

Installation and Setup

Stickler currently has a basic workflow. First, install and setup your Stickler repository.

jeremy@aramis:~ % sudo gem install stickler
  ============================================================

  Thank you for installing Stickler!

  * Create a new stickler repository:
      stickler setup /path/to/repo

  * Look at the help:
      stickler help

  ============================================================
Successfully installed stickler-0.1.0
1 gem installed
Installing ri documentation for stickler-0.1.0...
Installing RDoc documentation for stickler-0.1.0...

jeremy@aramis:~ % stickler setup /tmp/stickler
created repository root /tmp/stickler
created directory /tmp/stickler/gems
created directory /tmp/stickler/log
created directory /tmp/stickler/specifications
created directory /tmp/stickler/dist
created directory /tmp/stickler/cache
copied in default configuration to /tmp/stickler/stickler.yml
Setting up sources
 * loading http://gems.rubyforge.org/

A Stickler repository is a specific directory that holds all the gem files that are managed by Stickler for your personal distribution. In this case, Stickler is now installed as a gem and we have setup a repository in /tmp/stickler.

Populating the Repository

A repository is great, and completely useless unless it is populated with gems to distribute. So that is what we will do. The best way to operate on a Stickler repository is to be in it. You can also use the --directory commandline option on every command, but that can get old. Therefore, we’ll operate from the root of the repository.

cd /tmp/stickler
First lets look at the state of the repository using the info command. 
jeremy@aramis:/tmp/stickler % stickler info
Setting up sources
 * loading http://gems.rubyforge.org/

Upstream Sources
----------------

  http://gems.rubyforge.org/ : 16651 gems available
                             : 0 gems existing

Configured gems (in stickler.yml)
---------------------------------

Existing gems
-------------
Yes, that definitely is empty. We do a fair number of merb apps, so lets add merb to our repository. It is a 2 step process to add a gem to the repository via the commandline. We first pick the requirement operator, and then the version. In our case, we want to stick with a specific version of merb so we’ll add merb, = 0.9.8 to our repository 
jeremy@aramis:/tmp/stickler % stickler add gem merb
Setting up sources
 * loading http://gems.rubyforge.org/

You need to pick the merb Requirement to configure Stickler.
This involves picking one of the following Requirement operators
See http://docs.rubygems.org/read/chapter/16#page74 for operator info.

You need to (1) pick an operator and (2) pick a requirement.
The most common operators are >=, > and ~>
1. =  Equals version
2. != Not equal to version
3. >  Greater than version
4. <  Less than version
5. >= Greater than or equal to
6. <= Less than or equal to
7. ~> Approximately greater than
(1) Pick an operator ? 1

Now to pick a requirement.  Based upon your chosen operator '=',
These are the available version of the merb gem.
1. = 0.9.8   7. = 0.9.2   13. = 0.4.1  19. = 0.3.0  25. = 0.0.6
2. = 0.9.7   8. = 0.5.3   14. = 0.4.0  20. = 0.2.0  26. = 0.0.5
3. = 0.9.6   9. = 0.5.2   15. = 0.3.7  21. = 0.1.0  27. = 0.0.4
4. = 0.9.5   10. = 0.5.1  16. = 0.3.4  22. = 0.0.9  28. = 0.0.3
5. = 0.9.4   11. = 0.5.0  17. = 0.3.3  23. = 0.0.8  29. = 0.0.2
6. = 0.9.3   12. = 0.4.2  18. = 0.3.1  24. = 0.0.7  30. = 0.0.1
(2) Pick a requirement ? 1

Resolving gem dependencies for merb (= 0.9.8, runtime) ...
Adding ParseTree-2.1.1-x86-mswin32
Adding ruby2ruby-1.1.9
Adding RubyInline-3.7.0
Adding ParseTree-2.2.0
Adding merb-action-args-0.9.8
Adding merb-assets-0.9.8
Adding highline-1.4.0
Adding diff-lcs-1.1.2
Adding templater-0.3.2
Adding merb-gen-0.9.8
Adding haml-2.0.3
Adding merb-haml-0.9.8
Adding builder-2.1.2
Adding merb-builder-0.9.8
Adding mailfactory-1.4.0
Adding merb-mailer-0.9.8
Adding merb-parts-0.9.8
Adding merb-cache-0.9.8
Adding merb-slices-0.9.8
Adding merb-jquery-0.9.8
Adding extlib-0.9.7
Adding abstract-1.0.0
Adding erubis-2.6.2
Adding json_pure-1.1.3
Adding rubyforge-1.0.0
Adding rake-0.8.3
Adding hoe-1.8.0
Adding rspec-1.1.8
Adding rack-0.4.0
Adding mime-types-1.15
Adding hpricot-0.6.161-java
Adding hpricot-0.6.161
Adding hpricot-0.6-x86-mswin32
Adding thor-0.9.6
Adding merb-core-0.9.8
Adding merb-helpers-0.9.8
Adding merb-more-0.9.8
Adding mongrel-1.1.5-java
Adding mongrel-1.1.1-java
Adding daemons-1.0.10
Adding fastthread-1.0.1
Adding fastthread-1.0.1-x86-mswin32
Adding mongrel-1.1.5
Adding mongrel-1.1.3-x86-mswin32
Adding mongrel-1.1.5-x86-mswin32-60
Adding mongrel-1.1.2-x86-mswin32
Adding gem_plugin-0.2.3
Adding cgi_multipart_eof_fix-2.5.0
Adding mongrel-1.1.5-x86-mingw32
Adding merb-0.9.8
Wow, that’s a lot of gems added with for merb. And this is more than would be normally installed if you did a gem install merb. Stickler assumes you are redistributing gems, and as such gets every available platform for a gem when you add it to the repository. And then it recurses through all runtime and development dependencies. And now when we look at the repository state we see: 
Setting up sources
 * loading http://gems.rubyforge.org/

Upstream Sources
----------------

  http://gems.rubyforge.org/ : 16652 gems available
                             : 50 gems existing

Configured gems (in stickler.yml)
---------------------------------

merb : = 0.9.8

Existing gems
-------------

ParseTree-2.1.1-x86-mswin32
ParseTree-2.2.0
RubyInline-3.7.0
abstract-1.0.0
builder-2.1.2
cgi_multipart_eof_fix-2.5.0
daemons-1.0.10
diff-lcs-1.1.2
erubis-2.6.2
extlib-0.9.7
fastthread-1.0.1
fastthread-1.0.1-x86-mswin32
gem_plugin-0.2.3
haml-2.0.3
highline-1.4.0
hoe-1.8.0
hpricot-0.6-x86-mswin32
hpricot-0.6.161
hpricot-0.6.161-java
json_pure-1.1.3
mailfactory-1.4.0
merb-0.9.8
merb-action-args-0.9.8
merb-assets-0.9.8
merb-builder-0.9.8
merb-cache-0.9.8
merb-core-0.9.8
merb-gen-0.9.8
merb-haml-0.9.8
merb-helpers-0.9.8
merb-jquery-0.9.8
merb-mailer-0.9.8
merb-more-0.9.8
merb-parts-0.9.8
merb-slices-0.9.8
mime-types-1.15
mongrel-1.1.1-java
mongrel-1.1.2-x86-mswin32
mongrel-1.1.3-x86-mswin32
mongrel-1.1.5
mongrel-1.1.5-java
mongrel-1.1.5-x86-mingw32
mongrel-1.1.5-x86-mswin32-60
rack-0.4.0
rake-0.8.3
rspec-1.1.8
ruby2ruby-1.1.9
rubyforge-1.0.0
templater-0.3.2
thor-0.9.6

There is a difference between configured gems and existing gems. Configured gems are those you have specifically requested to be in the repository. Existing gems are those that are added to support the configured gems and the configured gems themselves.

Adding Additional Sources

Part of the reason for Stickler is to merge upstream gem repositories into your own internal repository. Say you use the github repository and want to add a gem or two from it to your internal system.

jeremy@aramis:/tmp/stickler % stickler add source http://gems.github.com
Setting up sources
 * loading http://gems.rubyforge.org/
 * loading http://gems.github.com/
http://gems.github.com added to sources
That was easy enough. Now lets add a gem from github and watch the dependencies roll in. 
jeremy@aramis:/tmp/stickler % stickler add gem aniero-tire_swing
Setting up sources
 * loading http://gems.rubyforge.org/
 * loading http://gems.github.com/

You need to pick the aniero-tire_swing Requirement to configure Stickler.
This involves picking one of the following Requirement operators
See http://docs.rubygems.org/read/chapter/16#page74 for operator info.

You need to (1) pick an operator and (2) pick a requirement.
The most common operators are >=, > and ~>
1. =  Equals version
2. != Not equal to version
3. >  Greater than version
4. <  Less than version
5. >= Greater than or equal to
6. <= Less than or equal to
7. ~> Approximately greater than
(1) Pick an operator ? =

Now to pick a requirement.  Based upon your chosen operator '=',
These are the available version of the aniero-tire_swing gem.
1. = 0.0.3  2. = 0.0.2
(2) Pick a requirement ? 1

Resolving gem dependencies for aniero-tire_swing (= 0.0.3, runtime) ...
Adding polyglot-0.2.3
Adding treetop-1.2.4
Adding attributes-5.0.1
Adding activesupport-2.1.1
Adding aniero-tire_swing-0.0.3
You can see that aneiro-tire_swing was added from github, and it also resolved the dependencies to gems that reside on rubyforge. Looking at info shows us: 
jeremy@aramis:/tmp/stickler % stickler info
Setting up sources
 * loading http://gems.rubyforge.org/
 * loading http://gems.github.com/

Upstream Sources
----------------

  http://gems.rubyforge.org/ : 16652 gems available
                             : 54 gems existing
     http://gems.github.com/ : 2463 gems available
                             : 1 gems existing

Configured gems (in stickler.yml)
---------------------------------

aniero-tire_swing : = 0.0.3
merb : = 0.9.8
...

Batch Addition

It can get repetitive to manually add gem after gem from the commandline. When you know exactly what you need, crack open the stickler.yml file and add a few gems. For instance, lets say that we are in the process of updating all our rails apps to 2.1. In the meantime we want to make sure we do not inadvertently update our production systems to 2.1. They are still at 2.0.2. So we add all the rails gems at the 2.0.2 level to the stickler.yml file in the repository.

---
downstream_source: http://gems.example.com/
sources:
- http://gems.rubyforge.org/
- http://gems.github.com/
gems:
  aniero-tire_swing:
  - = 0.0.3
  merb:
  - = 0.9.8
  activerecord: = 2.0.2
  activesupport: = 2.0.2
  actionmailer: = 2.0.2
  actionpack: = 2.0.2
  activeresource: = 2.0.2
  rails: = 2.0.2

We can now use the sync command to add all of them in at once.

jeremy@aramis:/tmp/stickler % stickler sync
Setting up sources
 * loading http://gems.rubyforge.org/
 * loading http://gems.github.com/

Making sure that all gems listed in configuration are available

Resolving gem dependencies for rails (= 2.0.2, runtime) ...
Adding activerecord-2.0.2
Adding actionpack-2.0.2
Adding actionmailer-2.0.2
Adding activesupport-2.0.2
Adding activeresource-2.0.2
Adding rails-2.0.2
Resolving gem dependencies for activerecord (= 2.0.2, runtime) ...
Resolving gem dependencies for activeresource (= 2.0.2, runtime) ...
Resolving gem dependencies for aniero-tire_swing (= 0.0.3, runtime) ...
Resolving gem dependencies for actionpack (= 2.0.2, runtime) ...
Resolving gem dependencies for actionmailer (= 2.0.2, runtime) ...
Resolving gem dependencies for merb (= 0.9.8, runtime) ...
Resolving gem dependencies for activesupport (= 2.0.2, runtime) ...

You can see that it added in all the gems in and then it also made sure that aneiro-tire_swing and merb were also synced up. If you need to rebuild the entire repository from scratch, use stickler sync --rebuild and it will wipe out the current gems and specifications in the repo and download them again.

Distribution of Gems

Populating the repository is only good if you can get your internal systems to utilize the repo. This is where the generate commands come into play. The generate index command does exactly what a gem generate_index does, but it uses your repository as the base.

Setting up sources
 * loading http://gems.rubyforge.org/
 * loading http://gems.github.com/
Generating rubygems index in /private/tmp/stickler/dist
Loading 61 gems from /private/tmp/stickler/dist
...............WARNING:  Skipping misnamed gem: /private/tmp/stickler/dist/gems/fastthread-1.0.1-x86-mswin32.gem => fastthread-1.0.1-x86-mswin32 (fastthread-1.0.1-mswin32)
.....WARNING:  Skipping misnamed gem: /private/tmp/stickler/dist/gems/hpricot-0.6-x86-mswin32.gem => hpricot-0.6-x86-mswin32 (hpricot-0.6-mswin32)
WARNING:  Skipping misnamed gem: /private/tmp/stickler/dist/gems/hpricot-0.6.161-java.gem => hpricot-0.6.161-java (hpricot-0.6.161-jruby)
..................WARNING:  Skipping misnamed gem: /private/tmp/stickler/dist/gems/mongrel-1.1.1-java.gem => mongrel-1.1.1-java (mongrel-1.1.1-jruby)
WARNING:  Skipping misnamed gem: /private/tmp/stickler/dist/gems/mongrel-1.1.2-x86-mswin32.gem => mongrel-1.1.2-x86-mswin32 (mongrel-1.1.2-mswin32)
WARNING:  Skipping misnamed gem: /private/tmp/stickler/dist/gems/mongrel-1.1.3-x86-mswin32.gem => mongrel-1.1.3-x86-mswin32 (mongrel-1.1.3-i386-mswin32)
....WARNING:  Skipping misnamed gem: /private/tmp/stickler/dist/gems/ParseTree-2.1.1-x86-mswin32.gem => ParseTree-2.1.1-x86-mswin32 (ParseTree-2.1.1-i386-mswin32)
............
Loaded all gems
Generating quick index gemspecs for 54 gems
......................................................
Complete
Generating specs index
Generating latest specs index
Generating quick index
Generating latest index
Generating Marshal master index
Generating YAML master index for 54 gems (this may take a while)
......................................................
Complete
Compressing indicies
The generated, distributable gem repository based upon your stickler repository is generated in the dist sub directory in your repository. This directory can be rsynced to a web server in your infrastructure, or run a web server on this machine and point its document root, or a directory alias to the dist directory.

An extra item you may do is globally configure your rubygems installations to automatically use your internal gem server as the canonical source instead of http://gems.rubyforge.org/. Stickler provides a quick command to generate this top level configuration, and the file contains the information on where to put it.

jeremy@aramis:/tmp/stickler % stickler generate sysconfig > gemrc
Setting up sources
 * loading http://gems.rubyforge.org/
 * loading http://gems.github.com/
Generating configuration to stdout

jeremy@aramis:/tmp/stickler % cat gemrc
#
# This is the system wide configuration to be used by
# rubygem clients that install gems from the repository
# located at :
#
#   http://gems.example.com/
#
# On Unix like machines install in
#
#   /etc/gemrc
#
# On Windows machines install in
#
#   C:\Documents and Settings\All Users\Application Data\gemrc
#
---
:sources:
- http://gems.example.com/

Conclusions

Hopefully you can find Stickler useful in your infrastructure. Please let me know about features and bugs.

Comments

blog comments powered by Disqus