Rubicon - a testsuite for Ruby ============================== This is **Rubicon**, a test suite for Ruby. Rubicon was originally written by Dave Thomas & Andy Hunt (for more info see the History_ section below). Now Rubicon is maintained as the project rubytests_ at RubyForge_. Rubicon provides testcases for the core Ruby language and for the built-in classes. We try to track the evolution of Ruby as it is implicitly specified by the C-implementation of the language. Currently (2005-01-18) Ruby 1.8.2 passes the Rubicon testsuite with very few errors (no errors at all on FreeBSD). There are still some errors on other platforms, but we are working on resolving these issues (see the Issues_ section below). Rubicon is far from complete: more tests need to be written, and existing tests need to be refined. If you want to help improve Rubicon, see the Contributing_ section below. Rubicon is also used by the team working on a Java implementation of Ruby (see http://jruby.sourceforge.net/). Requirements for running Rubicon -------------------------------- In order to use Rubicon you need: - a Ruby interpreter to test, e.g. Ruby_ 1.8.2 or JRuby_ version 0.8.0. - the `Test::Unit`_ framework. Recent versions of Ruby include Test::Unit, but if you want to test an older version of Ruby or a new fancy Ruby interpreter written in Haskell :) you have to make sure you have a working copy of Test::Unit. - a copy of this Rubicon testsuite. - a C compiler and a "make" program. The C compiler is used to build a couple of "utilities" invoked by the Rubicon tests. The "make" program is not strictly necessary: it is just used to build the utilities (located in the ``util`` subdirectory). Rubicon is written to be able to test different versions of Ruby. Recently much focus has been on getting Ruby 1.8.2 to work. Some older versions should also work (maybe with more errors). But maintaining a testsuite for a "moving target" like Ruby can be a bit tricky sometimes. Rubicon can be run on different OS-es. It has been used to test Ruby 1.8.2 on at least Linux, FreeBSD, Solaris, Mac OS-X and Windows-XP. Some of the tests are of course OS-dependent, so the testsuite give slightly different results on the different platforms. Getting Rubicon --------------- You can get Rubicon in a number of ways: - via CVS from the archives at rubyforge.org:: $ cvs -d :pserver:anonymous@rubyforge.org:/var/cvs/rubytests login $ cvs -d :pserver:anonymous@rubyforge.org:/var/cvs/rubytests co rubicon If you need more details about how to setup CVS access, see the "CVS Repository" link on the rubytests_ page. - fetching a release from the "Files" tab at rubytests_. The latest release may be a bit behind the "bleeding edge" in the CVS archives, but if you want a well-defined version, this may be the right choice. - getting a tar-file snapshot via the ViewCvs interface to the CVS archives. Look at the bottom of the page at: http://rubyforge.org/cgi-bin/viewcvs.cgi/rubicon/?cvsroot=rubytests Installation ------------ Once you have downloaded and unpacked the Rubicon files, you don't need to do anything more. Just start using it. Structure --------- The tests for the builtin classes/modules are in the subdirectory ``builtin``, and those for the language are in ``language``. Over time, different directories may be added with additional sets of tests (networking, for example). In order to run the tests, we need a small amount of support code, written in C. This is in the ``util`` directory. Running ------- There are two makefiles in Rubicon: one named ``GNUMakefile`` and another named ``Makefile``. The ``GNUMakefile`` should work on Linux and other UN*X variants. If your ``make`` program is not GNU Make (for example on Solaris or FreeBSD), you have to explicitly specify the name of the makefile. The ``Makefile`` is for Windows. In summary:: $ make ARGUMENTS # UN*X & GNU make $ make -f GNUMakefile ARGUMENTS # UN*X & non-GNU make $ nmake ARGUMENTS # Windows & Visual Studio In order to run the Rubicon tests, you have do do: 1) build the "utilities" in the ``util`` directory:: $ make util cc util/checkstat.c -o util/checkstat cc util/test_touch.c -o util/test_touch 2) choose one of the possible "entry points" to the tests:: # all Rubicon tests $ ruby AllTests.rb # all in the "builtin" directory $ cd builtin $ ruby AllBuiltinTests.rb # all in the "language" directory $ cd language $ ruby AllLanguageTests.rb # individual testcase file $ cd builtin $ ruby Test.rb # e.g. TestArray.rb # individual testcase file $ cd language $ ruby Test.rb 3) choose a Ruby interpreter to test. The simplest way is to make sure the desired version of Ruby is in your ``PATH`` (as in the example above). But but you can also specify the Ruby interpreter with an explicit path, like:: $ /somewhere/ruby-1.8.5/bin/ruby AllTests.rb ... $ /somewhere/ruby-1.8.6/bin/ruby AllTests.rb This way you will be able to easily test different Ruby versions without having to adjust the PATH. The testcases that spawn another Ruby instance makes sure that they start **the same** Ruby interpreter as the one used at the top level. 4) You can also run individual tests in a testcase file by giving the name of the test method as an argument on the command line:: $ ruby TestBignum.rb test_OR Mailing lists ------------- There are two mailing lists for Rubicon at RubyForge: - "rubytests-devel" is for general discussions about Rubicon. - "rubytests-commit" is a readonly list. For each CVS commit to the Rubicon archives, a mail is sent to this list. Look at the "Mailing Lists" link on the rubytests_ page for more info about these mailing lists. Contributing ------------ If you want to help improve the tests in Rubicon, join the mailing list where discussions about Rubicon take place. If you want to submit changes, we would appreciate context diffs (e.g. produced with ``cvs diff -u``). If you are (or will become) a regular contributor you should also become a member of the rubytests_ project at RubyForge_. That way you can get write access to the CVS archives, and commit changes yourself. There are many things that can be done to improve Rubicon: - add tests for methods that are not currently tested - strengthen existing tests - run the tests on more platforms, and adjust the testcases if necessary - ... Environmental Tests ------------------- *TODO: this section need to be updated, and expanded ...* Some tests depend on the environment in which they run. Guard these with appropriate conditions: - if your test depends on the Ruby version, using the Version class methods. Always passing in either a string or a range of strings in the form "1.6.2". Version is defined in ``rubicon.rb``. :: Version.in("1.6.3".."1.6.6") do # stuff that is meant to work in Ruby versions # 1.6.3 to 1.6.6 end Version.greater_than("1.6.2") do # code that depends on ruby version greater than 1.6.2 end - if your test depends on the operating system type, use the comparison classes. There's a class defined for each target operating system and environment (for example, Linux has one class, and Windows has MsWin32, Cygwin, and MinGW). These classes define class methods to allow you to include and exclude blocks of code depending on the OS that Rubicon is running under. :: MsWin32.only do # stuff for MsWin only end Windows.or_variant do # all Windows variants end Linux.dont do # don't run this under Linux end Have a look near the top of ``rubicon.rb`` for details. Issues ------ - what is correct floating point behaviour? Linux and FreeBSD differ. Should this "shine through" at the Ruby level, or should Ruby attempt to work the same way on different platforms? - some time-related tests now depend on the timezone of the used machine. The test fails if you are in (at least) some parts of the US. - *TODO: describe platform issues ...* - *TODO: need to add more here ...* Other Ruby testsuites --------------------- We believe that Rubicon still is the most comprehensive testsuite for the Ruby language and core libraries. But there some other tests too: - in the CVS archive for Ruby itself, there is a file ``sample/test.rb``. This is the file that is used when the ``make test`` command is used. - in the CVS archive for Ruby itself, there is a ``test`` subdirectory with a lot of tests. It is invoked by the ``make test-all`` command at the top level. History ------- Rubicon was originally written by Dave Thomas & Andy Hunt (http://www.pragmaticprogrammer.com) and was for a long time the authoritative test suite for the Ruby language. Rubicon was then maintained at ruby-lang.org alongside Ruby itself. These archives still exist has not been touched for a couple of years now. On a page (still accessible) at www.rubycentral.com an explanation is given for the name: *Rubicon is a set of tests for Ruby itself. (The name comes either from Ruby Confidence tests, or from the fact that once you're passed the Rubicon, the world is at your feet.)* November, 2000 They also said: *Rubicon is a set of unit tests for the Ruby interpreter and its libraries. Written using RubyUnit, the tests run automatically, allowing you to verify the correct installation of new versions of Ruby. We found this process to be invaluable while we were writing Programming Ruby, as it allowed us to check out changes in the CVS releases of Ruby with very little effort.* The current project rubytests_ at RubyForge_ is an attempt to revive Rubicon. When the project was started at RubyForge, Chad Fowler wrote the following in the README file: *We are now starting a renewed effort to bring rubicon up to date with Ruby 1.8+,* [...] Having Rubicon as a project at RubyForge has several advantages. We get the whole infrastructure at RubyForge_ for free: CVS-archives, user-accounts, mailing-lists, web-pages, etc. After Rubicon was moved to RubyForge in mars 2004, the tests were ported to `Test::Unit`_. Rubicon has also been updated to work with Ruby 1.8.2. Currently (2005-01-18) Ruby 1.8.2 passes the Rubicon testsuite with very few errors (no errors at all on FreeBSD). .. _Ruby: http://www.ruby-lang.org/ .. _JRuby: http://jruby.sourceforge.net/ .. _rubytests: http://rubyforge.org/projects/rubytests/ .. _RubyForge: http://rubyforge.org/ .. _`Test::Unit`: http://www.ruby-doc.org/stdlib/libdoc/test/unit/rdoc/index.html .. meta:: :description: Rubicon - a testsuite for Ruby :keywords: ruby, rubytests, testsuite