Rubicon - a testsuite for Ruby

This is Rubicon, a test suite for Ruby.

Rubicon was originally written by Dave Thomas & Andy Hunt (for more info see the History section below). Now Rubicon is maintained as the project rubytests at RubyForge.

Rubicon provides testcases for the core Ruby language and for the built-in classes. We try to track the evolution of Ruby as it is implicitly specified by the C-implementation of the language. Currently (2005-01-18) Ruby 1.8.2 passes the Rubicon testsuite with very few errors (no errors at all on FreeBSD). There are still some errors on other platforms, but we are working on resolving these issues (see the Issues section below).

Rubicon is far from complete: more tests need to be written, and existing tests need to be refined. If you want to help improve Rubicon, see the Contributing section below.

Rubicon is also used by the team working on a Java implementation of Ruby (see http://jruby.sourceforge.net/).

Requirements for running Rubicon

In order to use Rubicon you need:

Rubicon is written to be able to test different versions of Ruby. Recently much focus has been on getting Ruby 1.8.2 to work. Some older versions should also work (maybe with more errors). But maintaining a testsuite for a "moving target" like Ruby can be a bit tricky sometimes.

Rubicon can be run on different OS-es. It has been used to test Ruby 1.8.2 on at least Linux, FreeBSD, Solaris, Mac OS-X and Windows-XP. Some of the tests are of course OS-dependent, so the testsuite give slightly different results on the different platforms.

Getting Rubicon

You can get Rubicon in a number of ways:

Installation

Once you have downloaded and unpacked the Rubicon files, you don't need to do anything more. Just start using it.

Structure

The tests for the builtin classes/modules are in the subdirectory builtin, and those for the language are in language. Over time, different directories may be added with additional sets of tests (networking, for example).

In order to run the tests, we need a small amount of support code, written in C. This is in the util directory.

Running

There are two makefiles in Rubicon: one named GNUMakefile and another named Makefile. The GNUMakefile should work on Linux and other UN*X variants. If your make program is not GNU Make (for example on Solaris or FreeBSD), you have to explicitly specify the name of the makefile. The Makefile is for Windows. In summary:

$ make ARGUMENTS                      # UN*X & GNU make

$ make -f GNUMakefile ARGUMENTS       # UN*X & non-GNU make

$ nmake ARGUMENTS                     # Windows & Visual Studio

In order to run the Rubicon tests, you have do do:

  1. build the "utilities" in the util directory:

    $ make util
    cc     util/checkstat.c   -o util/checkstat
    cc     util/test_touch.c   -o util/test_touch
    
  2. choose one of the possible "entry points" to the tests:

    # all Rubicon tests
    $ ruby AllTests.rb
    
    # all in the "builtin" directory
    $ cd builtin
    $ ruby AllBuiltinTests.rb
    
    # all in the "language" directory
    $ cd language
    $ ruby AllLanguageTests.rb
    
    # individual testcase file
    $ cd builtin
    $ ruby Test<classname>.rb       # e.g. TestArray.rb
    
    # individual testcase file
    $ cd language
    $ ruby Test<feature>.rb
    
  3. choose a Ruby interpreter to test. The simplest way is to make sure the desired version of Ruby is in your PATH (as in the example above). But but you can also specify the Ruby interpreter with an explicit path, like:

    $ /somewhere/ruby-1.8.5/bin/ruby AllTests.rb
    ...
    $ /somewhere/ruby-1.8.6/bin/ruby AllTests.rb
    

    This way you will be able to easily test different Ruby versions without having to adjust the PATH. The testcases that spawn another Ruby instance makes sure that they start the same Ruby interpreter as the one used at the top level.

  4. You can also run individual tests in a testcase file by giving the name of the test method as an argument on the command line:

    $ ruby TestBignum.rb test_OR
    

Mailing lists

There are two mailing lists for Rubicon at RubyForge:

Look at the "Mailing Lists" link on the rubytests page for more info about these mailing lists.

Contributing

If you want to help improve the tests in Rubicon, join the mailing list where discussions about Rubicon take place. If you want to submit changes, we would appreciate context diffs (e.g. produced with cvs diff -u).

If you are (or will become) a regular contributor you should also become a member of the rubytests project at RubyForge. That way you can get write access to the CVS archives, and commit changes yourself.

There are many things that can be done to improve Rubicon:

Environmental Tests

TODO: this section need to be updated, and expanded ...

Some tests depend on the environment in which they run. Guard these with appropriate conditions:

Issues

Other Ruby testsuites

We believe that Rubicon still is the most comprehensive testsuite for the Ruby language and core libraries. But there some other tests too:

History

Rubicon was originally written by Dave Thomas & Andy Hunt (http://www.pragmaticprogrammer.com) and was for a long time the authoritative test suite for the Ruby language. Rubicon was then maintained at ruby-lang.org alongside Ruby itself. These archives still exist has not been touched for a couple of years now.

On a page (still accessible) at www.rubycentral.com an explanation is given for the name:

Rubicon is a set of tests for Ruby itself. (The name comes either from Ruby Confidence tests, or from the fact that once you're passed the Rubicon, the world is at your feet.)

November, 2000

They also said:

Rubicon is a set of unit tests for the Ruby interpreter and its libraries. Written using RubyUnit, the tests run automatically, allowing you to verify the correct installation of new versions of Ruby. We found this process to be invaluable while we were writing Programming Ruby, as it allowed us to check out changes in the CVS releases of Ruby with very little effort.

The current project rubytests at RubyForge is an attempt to revive Rubicon. When the project was started at RubyForge, Chad Fowler wrote the following in the README file:

We are now starting a renewed effort to bring rubicon up to date with Ruby 1.8+, [...]

Having Rubicon as a project at RubyForge has several advantages. We get the whole infrastructure at RubyForge for free: CVS-archives, user-accounts, mailing-lists, web-pages, etc.

After Rubicon was moved to RubyForge in mars 2004, the tests were ported to Test::Unit. Rubicon has also been updated to work with Ruby 1.8.2. Currently (2005-01-18) Ruby 1.8.2 passes the Rubicon testsuite with very few errors (no errors at all on FreeBSD).