If you install Rails 1.2.4 on your system, then running any Rails project that was built with 1.2.3 will magically use version 1.2.4 even if you have explicitly declared RAILS_GEM_VERSION = ' 1.2.3' in your environment.rb. This may only happen in environments that load boot.rb explicitly; I noticed it when running our app under mongrel but it seemed to work ok in unit tests.
The workaround is simple, if maddening: you must put a space character at the beginning of the following line in your environment.rb:
RAILS_GEM_VERSION = '1.2.3' unless defined? RAILS_GEM_VERSION
This bug is present in both Rails 1.2.3 and Rails 1.2.4 (meaning that projects generated with Rails 1.2.4 have the same buggy code in boot.rb and therefore the line in the generated environment.rb will be missed).
I've attached a patch with tests.