[PATCH] Vendor Everything

+1! Surely this is becoming a standard idiom -- might as well make it easier with this patch.

agreed

Marcel's patch from Rails Core mailing list

+1 tally-ho

One problem - how does this handle multiple platforms? In other words, if my app is developed and used on different platforms? The same version of a gem can have multiple platforms, which are packaged with different binaries. If you freeze a gem version for one platform into vendor, the app would not run on another platform that is incompatible.

Replying to thewoolleyman:

One problem - how does this handle multiple platforms? In other words, if my app is developed and used on different platforms? The same version of a gem can have multiple platforms, which are packaged with different binaries. If you freeze a gem version for one platform into vendor, the app would not run on another platform that is incompatible.

Most gems don't need to be compiled and are cross platform. And if it does, don't freeze it. 80/20 rule.

Replying to josh:

Most gems don't need to be compiled and are cross platform. And if it does, don't freeze it. 80/20 rule.

So should the name of this ticket be changed to "Vendor Everything that is Pure Ruby"? ;)

Better?

j/k

What gems are you exactly talking about?

Like hpricot? If the gem requires C compiling, some who argue that the gem would have better performance if compiled on the server.

There's lots of gems that have ms-windows platforms. Platform-related discussions are frequent on the rubygems mailing list (and there's one regarding jruby in progress right now).

You could find out yourself by grepping the local gem specs on a windows machine to see what the platform is. As an unrelated experiment, I just installed all the gems off my mac onto a clean Windows XP rubygems installation. I'll try to see how many have a /mswin/ platform and report the results.

Also, I agree that it would probably be a good idea to compile any gem that needs compiling on the actual machine which will be running the gem. If you agree with this, then "Vendor Everything that is Pure Ruby" is actually pretty appropriate.

-- Chad

Sure, I agree, it only applies to ruby code. But I don't find it a huge problem.

The title was just named after Chris Wanstrath's popular post, "Vendor Everything".

Here's some empirical data. These are gem specs which were on my mac, and I then installed on Windows XP, grepped for a platform matching "win":

name: ParseTree name: RubyInline name: SwedishTV name: ZenTest name: actionmailer name: actionpack name: actionwebservice name: activerecord name: activesupport name: acts_as_searchable name: aws-s3 name: builder name: capistrano name: cgi_multipart_eof_fix name: cheat name: coderay name: daemons name: fastthread platform: mswin32 name: firebrigade_api name: fjson name: flexmock name: fxri name: fxruby platform: mswin32 name: gem_plugin name: heckle name: hpricot platform: mswin32 name: log4r name: meta_project name: mime-types name: mkrf name: money name: mongrel platform: mswin32 name: needle name: net-sftp name: net-ssh name: piston name: rails name: rc-rest name: rjb platform: mswin32 name: rspec name: ruby-doom name: ruby2ruby name: rutils name: s3-ruby name: s33r name: sources name: sqlite3-ruby platform: mswin32 name: syntax name: tinderbox name: win32-clipboard name: win32-dir platform: mswin32 name: win32-eventlog name: win32-file-stat platform: mswin32 name: win32-file-stat platform: mswin32 name: win32-process name: win32-sapi platform: mswin32 name: win32-sound name: windows-pr name: windows-pr name: windows-pr name: x10-cm17a platform: i386-mswin32 name: xml-simple

In addition, these were installed on mac but don't have a windows build available (and thus won't even work on Windows):

name: fastthread name: ferret name: rmagick name: termios name: x10-cm17a

So, there's not too many, but some popular ones, like hpricot, mongrel/fastthread, and sqlite. You could probably perform this operation on the entire rubygems repository, but it would be difficult, since the "gem specification" command can't be run against a remote repository.

Hmm. That didn't work too well, but you get the idea. Put in the newlines if you want...

Many of those gems listed are pure ruby but have some little modifications for windows. Most of contain a bat script to replace the shell script bundle. Take rails for example. The rails command uses the bat file on windows and the shell script on unix.

However all this doesn't make a difference. You aren't freezing utilities, you are freezing libraries. You can still load the actual library portion.

Its useless to freeze system utilities like piston or capistrano. Maybe not useless, but not the point.

This patch a simple solution to freeze ruby code in vendor/gems and it works in most cases.

I do think its worth investigating a solution that supports platform-specific gems when a rails app is first run.

E.g. if you link in 'hpricot 0.5' vs freezing it, then when you launch your app, and if hpricot 0.5 isn't available you are asked to install it, or the app quits with an error.

Auto-install will be available when its supported by RubyGems in the future.

Whatever our solution, we should consider the gems that have platform specific versions.

Similarly, if you develop your app on a Mac, and want to deploy to Tomcat on jruby... well, that's a funny coloured kettle of fish to consider oneday.

Perhaps this ticket is best developed in one of the existing gem solutions (http://errtheblog.com/post/2120), and then we integrate the gem code into core later, or link in as a dependency of rails gem.

I'll have a look into the missing code from the gemsonrails gem (http://drnicwilliams.com/2007/02/09/railsrally-2007-and-gemsonrails/#railsrally-gemsonrails) towards "complaining on missing gem versions that are linked by not frozen".

Anyone else who wants to look at the code via SVN its at http://rubyforge.org/scm/?group_id=2391

Easy peasy.

Nic

Coincidentally (or maybe not) the RubyGems developers seem to be ready to tackle the platform issue:

http://rubyforge.org/pipermail/rubygems-developers/2007-April/002733.html

http://tattle.rubygarden.org/

We should probably move this debate over to the mailing list so we don't clog the internet tubes to the trac server.

http://groups.google.com/group/rubyonrails-core/browse_thread/thread/e48c42e133ee2bf9/#

Still, this whole platform thing doesn't concern me, and I leave it up to you to PDI.

Idea still needs more work. Closing this ticket. If you are still back the idea, start up your own ticket (Please don't reopen this one).

Just for future reference and closure, can we have a summary of the technical issues that caused this ticket to be killed? Links to other external references/articles/blogs would be fine.

I'm just interested in the underlying validity of the whole "vendor everything" approach, and whether it's fundamentally impossible, or just not possible in the immediate timeframe.

If you don't want to bloat this ticket with more comments, feel free to email me at thewoolleyman@gmail.com.

Thanks! -- Chad

No problem Chad. The main article you're looking for is "err.the_blog.find_by_title('Vendor Everything')" (http://errtheblog.com/post/2120).