Requirements for running Rubicon

In order to use Rubicon you need:

• a Ruby interpreter to test, e.g. Ruby 1.8.2 or JRuby version 0.8.0.
• the Test::Unit framework. Recent versions of Ruby include Test::Unit, but if you want to test an older version of Ruby or a new fancy Ruby interpreter written in Haskell :) you have to make sure you have a working copy of Test::Unit.
• a copy of this Rubicon testsuite.
• a C compiler and a "make" program. The C compiler is used to build a couple of "utilities" invoked by the Rubicon tests. The "make" program is not strictly necessary: it is just used to build the utilities (located in the util subdirectory).

Rubicon is written to be able to test different versions of Ruby. Recently much focus has been on getting Ruby 1.8.2 to work. Some older versions should also work (maybe with more errors). But maintaining a testsuite for a "moving target" like Ruby can be a bit tricky sometimes.

Rubicon can be run on different OS-es. It has been used to test Ruby 1.8.2 on at least Linux, FreeBSD, Solaris, Mac OS-X and Windows-XP. Some of the tests are of course OS-dependent, so the testsuite give slightly different results on the different platforms.

Getting Rubicon

You can get Rubicon in a number of ways:

• via CVS from the archives at rubyforge.org:

  $ cvs -d :pserver:anonymous@rubyforge.org:/var/cvs/rubytests login
  $ cvs -d :pserver:anonymous@rubyforge.org:/var/cvs/rubytests co rubicon
  

  If you need more details about how to setup CVS access, see the "CVS Repository" link on the rubytests page.

• fetching a release from the "Files" tab at rubytests. The latest release may be a bit behind the "bleeding edge" in the CVS archives, but if you want a well-defined version, this may be the right choice.

• getting a tar-file snapshot via the ViewCvs interface to the CVS archives. Look at the bottom of the page at: http://rubyforge.org/cgi-bin/viewcvs.cgi/rubicon/?cvsroot=rubytests

Installation

Once you have downloaded and unpacked the Rubicon files, you don't need to do anything more. Just start using it.

Structure

The tests for the builtin classes/modules are in the subdirectory builtin, and those for the language are in language. Over time, different directories may be added with additional sets of tests (networking, for example).

In order to run the tests, we need a small amount of support code, written in C. This is in the util directory.

Running

There are two makefiles in Rubicon: one named GNUMakefile and another named Makefile. The GNUMakefile should work on Linux and other UN*X variants. If your make program is not GNU Make (for example on Solaris or FreeBSD), you have to explicitly specify the name of the makefile. The Makefile is for Windows. In summary:

$ make ARGUMENTS                      # UN*X & GNU make

$ make -f GNUMakefile ARGUMENTS       # UN*X & non-GNU make

$ nmake ARGUMENTS                     # Windows & Visual Studio

In order to run the Rubicon tests, you have do do:

1. build the "utilities" in the util directory:

   $ make util
   cc     util/checkstat.c   -o util/checkstat
   cc     util/test_touch.c   -o util/test_touch
   

2. choose one of the possible "entry points" to the tests:

   # all Rubicon tests
   $ ruby AllTests.rb
   
   # all in the "builtin" directory
   $ cd builtin
   $ ruby AllBuiltinTests.rb
   
   # all in the "language" directory
   $ cd language
   $ ruby AllLanguageTests.rb
   
   # individual testcase file
   $ cd builtin
   $ ruby Test<classname>.rb       # e.g. TestArray.rb
   
   # individual testcase file
   $ cd language
   $ ruby Test<feature>.rb
   

3. choose a Ruby interpreter to test. The simplest way is to make sure the desired version of Ruby is in your PATH (as in the example above). But but you can also specify the Ruby interpreter with an explicit path, like:

   $ /somewhere/ruby-1.8.5/bin/ruby AllTests.rb
   ...
   $ /somewhere/ruby-1.8.6/bin/ruby AllTests.rb
   

   This way you will be able to easily test different Ruby versions without having to adjust the PATH. The testcases that spawn another Ruby instance makes sure that they start the same Ruby interpreter as the one used at the top level.

4. You can also run individual tests in a testcase file by giving the name of the test method as an argument on the command line:

   $ ruby TestBignum.rb test_OR
   

Mailing lists

There are two mailing lists for Rubicon at RubyForge:

• "rubytests-devel" is for general discussions about Rubicon.
• "rubytests-commit" is a readonly list. For each CVS commit to the Rubicon archives, a mail is sent to this list.

Look at the "Mailing Lists" link on the rubytests page for more info about these mailing lists.

Contributing

If you want to help improve the tests in Rubicon, join the mailing list where discussions about Rubicon take place. If you want to submit changes, we would appreciate context diffs (e.g. produced with cvs diff -u).

If you are (or will become) a regular contributor you should also become a member of the rubytests project at RubyForge. That way you can get write access to the CVS archives, and commit changes yourself.

There are many things that can be done to improve Rubicon:

• add tests for methods that are not currently tested
• strengthen existing tests
• run the tests on more platforms, and adjust the testcases if necessary
• ...

Environmental Tests

TODO: this section need to be updated, and expanded ...

Some tests depend on the environment in which they run. Guard these with appropriate conditions:

• if your test depends on the Ruby version, using the Version class methods. Always passing in either a string or a range of strings in the form "1.6.2". Version is defined in rubicon.rb.

  Version.in("1.6.3".."1.6.6") do
    # stuff that is meant to work in Ruby versions
    # 1.6.3 to 1.6.6
  end
  
  Version.greater_than("1.6.2") do
    # code that depends on ruby version greater than 1.6.2
  end
  

• if your test depends on the operating system type, use the comparison classes. There's a class defined for each target operating system and environment (for example, Linux has one class, and Windows has MsWin32, Cygwin, and MinGW). These classes define class methods to allow you to include and exclude blocks of code depending on the OS that Rubicon is running under.

  MsWin32.only do
    # stuff for MsWin only
  end
  
  Windows.or_variant do
    # all Windows variants
  end
  
  Linux.dont do
    # don't run this under Linux
  end
  

  Have a look near the top of rubicon.rb for details.

Issues

• what is correct floating point behaviour? Linux and FreeBSD differ. Should this "shine through" at the Ruby level, or should Ruby attempt to work the same way on different platforms?
• some time-related tests now depend on the timezone of the used machine. The test fails if you are in (at least) some parts of the US.
• TODO: describe platform issues ...
• TODO: need to add more here ...

Other Ruby testsuites

We believe that Rubicon still is the most comprehensive testsuite for the Ruby language and core libraries. But there some other tests too:

• in the CVS archive for Ruby itself, there is a file sample/test.rb. This is the file that is used when the make test command is used.
• in the CVS archive for Ruby itself, there is a test subdirectory with a lot of tests. It is invoked by the make test-all command at the top level.

History

Rubicon was originally written by Dave Thomas & Andy Hunt (http://www.pragmaticprogrammer.com) and was for a long time the authoritative test suite for the Ruby language. Rubicon was then maintained at ruby-lang.org alongside Ruby itself. These archives still exist has not been touched for a couple of years now.

On a page (still accessible) at www.rubycentral.com an explanation is given for the name:

Rubicon is a set of tests for Ruby itself. (The name comes either from Ruby Confidence tests, or from the fact that once you're passed the Rubicon, the world is at your feet.)

November, 2000

They also said:

Rubicon is a set of unit tests for the Ruby interpreter and its libraries. Written using RubyUnit, the tests run automatically, allowing you to verify the correct installation of new versions of Ruby. We found this process to be invaluable while we were writing Programming Ruby, as it allowed us to check out changes in the CVS releases of Ruby with very little effort.

The current project rubytests at RubyForge is an attempt to revive Rubicon. When the project was started at RubyForge, Chad Fowler wrote the following in the README file:

We are now starting a renewed effort to bring rubicon up to date with Ruby 1.8+, [...]

Having Rubicon as a project at RubyForge has several advantages. We get the whole infrastructure at RubyForge for free: CVS-archives, user-accounts, mailing-lists, web-pages, etc.

After Rubicon was moved to RubyForge in mars 2004, the tests were ported to Test::Unit. Rubicon has also been updated to work with Ruby 1.8.2. Currently (2005-01-18) Ruby 1.8.2 passes the Rubicon testsuite with very few errors (no errors at all on FreeBSD).