[PATCH] Support plugin dependencies

This is a welcome patch!

First patch was missing new test files.

The recent addition of config.plugins will let you specify which plugins to load, and which order to load them in.

Replying to nzkoz:

The recent addition of config.plugins will let you specify which plugins to load, and which order to load them in.

IMHO, that's not a great solution for this problem. For one, it's not really DRY. It would mean that I would have to specify the ordering of my plugins in every single application that uses those plugins. When plugins are released, the authors would then have to rely on the developer specify the correct order in config.plugins. It may be a bit of an exaggeration, but it's almost like forcing the developer to specify all of the requires for an application in some config option, rather than the requires being defined where they should (i.e. in the actual files that use them).

Having over 70 plugins in a specific order in config.plugins, to me, is difficult to maintain. I like to keep information about dependencies local to where they actually occur. config.plugins is a step in the right direction because it gives us the ability to specify which plugins are loaded, however I don't think it is a solution to the plugin dependency problem. Obviously, my patch is opinionated, but don't we want to make it easy for developers :) Just pop in a couple of plugins from some vendor through svn:externals and bam, you're done! No configuration of dependency order... plugin nirvana? :)

Thoughts?

I'd start by seriously questioning why you feel you need 70(!) plugins in the first place. I'd love to see just a directory index of your vendor/plugins, just to get a sense how the granularity of your plugins. Initial feeling is that needing plugin dependencies is a smell of over-reliance on plugins, but I'll rest on further judgement for now.

David: I'd love to show you it (over 20,000 LOC), but we plan on releasing them to the Rails public soon enough and I don't want to spoil it here. I'd be glad to e-mail it privately to you.

Who do I send to, david@loudthinking.com?

Replying to obrie:

David: I'd love to show you it (over 20,000 LOC), but we plan on releasing them to the Rails public soon enough and I don't want to spoil it here. I'd be glad to e-mail it privately to you.

The specifics of what the plugins are called or what they don't aren't the really important part. I just can't see what you'd possibly need 70 plugins for, I mean, it's a lot... Beyond the 70 plugins, I can't see why they'd be so interlinked that the load order would matter that much.

While we want to make developers lives as easy as possible, we don't want to imply that writing an app with a hundred plugins is a normal situation either.

Replying to nzkoz:

The specifics of what the plugins are called or what they don't aren't the really important part. I just can't see what you'd possibly need 70 plugins for, I mean, it's a lot... Beyond the 70 plugins, I can't see why they'd be so interlinked that the load order would matter that much. While we want to make developers lives as easy as possible, we don't want to imply that writing an app with a hundred plugins is a normal situation either.

My friends, those are famous last words :) I can't see anyone need more than 720K of memory :P I'd imagine you can't see why you would need it or how they could be interlinked so much because you've never been in a situation that required so many plugins (I just reread this sentence.. just want to make sure you know no offense was meant in that sentence :)). What better way to show you why so many plugins are needed than show you what the plugins are and how they depend on the existence of other plugins? I could start going into the theory of plugins, but it seems like a hassle. By the way, I'm in the middle of submitting a paper to RailsConf 2007 on the theory of plugins :)

But anyway, I think having large-scale applications like MySpace and YouTube would absolutely end up resulting in developing over 100 plugins. Every time you write some code that could potentially be reused by another developer or in another part of the application, bam! That's a plugin. Just look at how many plugins are already on the wiki. I could be using plugins for ActionMailer, ActiveRecord, ActionPack, Unit::Test, Ruby core, etc. etc. the list goes on. Now don't get me wrong, not all of my plugins are small. Recent rake:stats showed between 20K-25K in library code split between 70 custom plugins + 10 vendor plugins. In every plugin we write, we see a potential use by others in the community. In addition, we don't want to pack too much into a plugin. Developers want to pick and choose what code goes into their application; it's why I created an alternative to the engines plugin. Why be forced with a gigantic plugin like engines when you could create 6 different plugins that achieve the same functionality, but give developers the choice as to which types of functionality they actually want included (assets automatically copied, app/models,controllers,component, plugin migrations, etc.).

I hate for this to become a discussion on why I need 70 plugins. I'm leaving that to my (hopefully) presentation at RailsConf. I'd like to focus more on how config.plugins is going to encourage lots of applications to add more configuration code when that configuration code could simply be replaced by descriptive dependency requirements in each plugin (similar to the type of descriptive technique used in this patch).

I actually meant to say that most of the configuration code could be replaced (in that last paragraph).

The specifics has to be part of the discussion. We don't add features to Rails without knowing the real-world usage patterns or implications. So plugin dependency is not happening before there's an established case showing that to be needed for doing something in what's deemed a proper way. It may well be that your plugin usage is considered appropriate, but before that's established it's premature to discuss ways to support that.

As I read your reply, I get even more convinced that we need to further inspect this use case. Statements like:

Every time you write some code that could potentially be reused by another developer or in another part of the application, bam! That's a plugin. 

That doesn't seem like entirely sound advice to me. You shouldn't abstract every thing that could potentially be reused. You should only abstract things that actually are reused. In other words, having a single case for something is often doubtful grounds for abstraction.

Second, you say "...or in another part of the application". I don't follow that? Plugins only make sense if they're shared between different applications. Not inside of the same one.

That doesn't seem like entirely sound advice to me. You shouldn't abstract every thing that could potentially be reused. You should only abstract things that actually are reused. In other words, having a single case for something is often doubtful grounds for abstraction.

I'm not necessarily saying abstraction. I'm saying moving the code to something that could be reused, not modifying the code so that it could be used for others. For example, I created a plugin called validates_xor_presence_of that validates only one of 2 or more columns are present. Now I'm only using that in one application, but I could see potential uses for it elsewhere. I'm just putting the code in it's own file and folder so that at some point when I have time, I can release it as a plugin that others can reuse.

Second, you say "...or in another part of the application". I don't follow that? Plugins only make sense if they're shared between different applications. Not inside of the same one.

You're right, I wasn't very clear on that. It goes along with what I said above. In my application, I use validates_xor_presence_of in 2 different models. There's almost no difference between putting that somewhere in my application code and putting it in a plugin. It requires perhaps 1 or 2 more lines of code. I might as well create it as a plugin that can be managed as a separate component which others can maintain outside of my current application. By bundling it in my application code, it means that whoever is maintaining the application also has to maintain the validates_xor_presence_of functionality. However, by putting it in a plugin, I create a separate trac so that people can submit bugs specifically for the validates_xor_presence_of plugin and maintain it completely separate from my core app.

Every validates_* or acts_* I create a plugin for it. They may be validates_as_email, acts_as_commentable, acts_as_rateable, acts_as_messageable, etc. Each these are common and can be put into plugins that are maintained outside of the application I am developing.

As a real-world example of some complicated plugin dependencies, I have a payment plugin. In a payment library, you have addresses (for credit cards), state machines (state of a transaction), credit card types (uses acts_as_enumerated), validations (credit card number can't change [validates_as_readonly], product costs [validates_numericality_of with :greater_than, :less_than, etc. constraints]), and others (boolean_attributes for doing something like battr_accessor, battr_reader, etc.). For testing, it uses a plugin called dry_validity_assertions that makes it easy to test the validity/invalidity of a record (more robust than it is in rails core).

So the payment library depends on the following plugins:

acts_as_addressable
acts_as_enumerated
acts_as_state_machine

validates_as_readonly
validates_numericality_of_constraints

boolean_attributes
dry_validity_assertions

Now some of these have their own dependencies. acts_as_state_machine relies on a plugin called dry_transaction_rollbacks that lets you simply call rollback in a transaction, and it'll get out without you having to do a begin-rescue around all blocks in which you want to do a rollback.

Then I write a paypal extension to the payment library, so that you can interact with paypal. Again, I don't want to force people to add paypal code to their application, so I write another plugin called pay_pal_payment which depends on the presence of the payment plugin. In addition, it uses a plugin called soap4ar which allows you to use ActiveRecord classes generated from the PayPal WSDL and use them to interact with the web service via SOAP. It also uses a plugin called kind_associations which lets paypal requests to have multiple has_one associations on the same table, without having to write conditions and such to differentiate each has_one from the other.

So now we have the pay_pal_payment plugin which has the following dependencies:

soap4ar
kind_associations

Now all of these plugins can be reused for other purposes by other people. In order to make it easier for other people who want to reuse the payment or pay_pal_payment library, the plugin dependencies are defined right in the init.rb. A few of the plugins have other dependencies themselves, but I won't list them here, you get the idea.

Then, we have plugins which let you define other plugins that add models to your application as skeletons (when you access, for example, the User model, not only will the user model from the user_skeleton plugin be loaded, but it'll also load the User model defined in your application, merging the two). Any plugin that wants to add models then, depends on this plugin called appable_plugins.

I mean, we have other stuff like a column filters plugin that filters content before it's saved to the database. If you want to create a column filter to be used by others, simply create a new plugin, add a dependency on the column_filters plugin, and now you've got an extension to that original plugin. Now when those two plugins are loaded, they're automatically loaded in the correct order because of the dependency states in in new filter's init.rb.

There are more examples, but I hope this has provided some sort of insight into a real-world example. I certainly haven't gone over everything, but it gives you an idea.

We're definitely not going to add this for rails 1.2. While this may sound facetious, I'm confident the best bet will be for you to release a plugin enabling this functionality.

I guess you'll have to call it aaaaaaaaa_dependencies or something... If it catches on, and develops some real-world use, we can consider it for inclusion in 2.0.

If you'd like to discuss this more, I suggest you jump on the rails core list

http://groups.google.com/group/rubyonrails-core

At a more fundamental level, why not release these libraries as gems? that way in addition to the dependencies, you get simplified installation / updates.

I know this hasn't gotten a lot of support, so I thought I would suggest another solution that tries to bridge the gap between plugins and gems. The patch I'm suggesting here modifies the initialization of plugins by:

1. Adding all lib paths to $LOAD_PATH first.
2. Then loading the init.rb for each plugin.

This solves some issues discussed here by, first and foremost, getting rid of the need for another require api. It also means that, whether something is in the plugins folder or in your gems folder, it can be loaded the same way.

So, for example, in my init.rb, I would do:

require 'plugin_with_dependency'

and then in lib/plugin_with_dependency.rb:

require 'dependent_plugin'

...rest of plugin code...

This would encourage plugin directory structures to be very similar to gems where you would place all of your main plugin code in lib/plugin_xyz.rb (and then require other files from there within the lib/plugin_xyz/ folder).

Alternative solution. [not applicable anymore; already being done in Edge]

Scratch the alternative solution... didn't realize something similar was already being done in edge with load_paths.