Ruby on Rails | Screencasts | Download | Documentation | Weblog | Community | Source

Ticket #3591 (closed defect: fixed)

Opened 4 years ago

Last modified 3 years ago

[PATCH] ActiveRecord::Base.remove_connection doesn't remove connection, corrupts connection cache

Reported by: simon.stapleton@gmail.com Assigned to: Jeremy Kemper <rails@bitsweat.net>
Priority: high Milestone: 1.1
Component: ActiveRecord Version: 1.0.0
Severity: critical Keywords: fd
Cc: f.svehla@gmail.com, tom@popdog.net

Description

Consider the following script:

require 'rubygems'
require 'active_record'

begin
  0.upto(1000) do |i|
    puts "Starting connection #{i}"
    ActiveRecord::Base.establish_connection(:adapter => "postgresql",
                                            :host => "localhost",
                                            :database => "database",
                                            :username => "user",
                                            :password => "pass")
    # Do something to the connection to make it actually fire up.
    puts "connected" if ActiveRecord::Base.connection.active?
    puts "Stopping connection #{i}"
    ActiveRecord::Base.remove_connection
  end
rescue
  puts $!.inspect
end

What this should do is fire up and then instantaneously close 1000 database connections. It should never have more than one connection open at a time. What it actually does, against a stock postgres install, is fire up 50 connections and then fail with a postgres error on the 51st.

It's a stupid example, but a valid example of what is breaking my application.

What's causing this is the following code, in

    def self.remove_connection(klass=self)
      conn = @@defined_connections[klass.name]
      @@defined_connections.delete(klass.name)
      @@connection_cache[Thread.current.object_id].delete(klass.name)
      active_connections.delete(klass.name)
      @connection = nil
      conn.config if conn
    end

Note the absolute lack of any closing of the connection, all we do is remove it from the memory caches. It appears that, for the postgres C adaptor at least, this does not cause the actual connection to be closed.

It seems that this is related to the known issues with C language extensions and memory reclamation, similar to the need to manually run GC.start when you're done with RMagick objects; indeed, if we add GC.start to the original script after the remove_connection, all is well.

I would suggest that remove_connection is modified either to explicitly close the connection (my preferred option, as this leaves nothing to luck, but requires a little work) or at least to trigger garbage collection after clearing its caches.

Simon

Attachments

disconnect_database_when_removing_connection.diff (3.8 kB) - added by tomafro on 01/24/06 17:24:01.
tomschanges+simonschanges.diff (4.2 kB) - added by simon.stapleton@gmail.com on 01/24/06 19:48:40.
tom's patch as attached before with corresponding postgresql bits
active_record_patch.diff (5.4 kB) - added by simon.stapleton@gmail.com on 02/03/06 11:27:01.
fix incorporating cache cleanup

Change History

01/24/06 14:38:39 changed by anonymous

  • cc set to tom@popdog.net.

01/24/06 16:19:34 changed by anonymous

  • cc changed from tom@popdog.net to f.svehla@gmail.com.

01/24/06 16:38:54 changed by tomafro

  • cc changed from f.svehla@gmail.com to f.svehla@gmail.com, tom@popdog.net.

01/24/06 17:21:19 changed by tomafro

  • summary changed from ActiveRecord::Base.remove_connection doesn't remove connection to [PATCH] ActiveRecord::Base.remove_connection doesn't remove connection.

This is a problem I've come across too, but (to my shame) ignored until now. I've written a simple patch that calls a new adapter method, disconnect!, on the connection when removing it.

I've written and tested implementations of this method for MySQL and SQL Server, but not for any of the other adapters (as I don't have installations to test against). To get it to work against postgresql, all that would be required is to implement disconnect! in the adapter. If you work out how to do this, please add it as another patch to this issue.

01/24/06 17:24:01 changed by tomafro

  • attachment disconnect_database_when_removing_connection.diff added.

01/24/06 19:48:40 changed by simon.stapleton@gmail.com

  • attachment tomschanges+simonschanges.diff added.

tom's patch as attached before with corresponding postgresql bits

01/25/06 06:20:18 changed by david

  • keywords set to fd.

02/03/06 11:27:01 changed by simon.stapleton@gmail.com

  • attachment active_record_patch.diff added.

fix incorporating cache cleanup

02/03/06 11:27:22 changed by simon.stapleton@gmail.com

  • summary changed from [PATCH] ActiveRecord::Base.remove_connection doesn't remove connection to [PATCH] ActiveRecord::Base.remove_connection doesn't remove connection, corrupts connection cache.

Also found another issue when working on this, as follows:

As it stands, when we remove a connection, it is only removed from the connection cache et al for the class for which remove_connection is called. Current release version does not cause a problem, as the connection itself is not closed, but now we're closing the connection, it leaves 'floating' connections that are not actually connected, and teh whole connection cache gets out of whack. Consider the following scenario:

We have a subclass of ActiveRecord::Base called ModelClass

First off, we create a connection to the database using ActiveRecord::Base.establish_connection. @@connection_cache for this thread now has one entry, for ActiveRecord::Base, pointing to our connection, as expected.

When we first access ModelClass we will modify the cache, and it will now contain entries for 'ActiveRecord::Base' and 'ModelClass', both of which point to the same connection.

Somewhere else in our application we call ActiveRecord::Base.remove_connection. This removes the entry in @@connection_cache for ActiveRecord::Base and closes the connection to the database, but leaves an entry for ModelClass pointing to the now defunct connection.

Later on, we access ModelClass again, and we get an ActiveRecord exception (in my case, wrapping a PGconn exception), again as expected - the model is not connected to a database

The problem is what to do here. Obviously, if we're dropping connections, we need to handle such exceptions, so what do we do? We are probably not in ModelClass itself, so what we do is the sensible thing : before our first database access we use something like this:

ActiveRecord::Base.connection rescue ActiveRecord::Base.establish_connection

This should force us to create a new connection to the database, and in fact is will do so. A subsequent test of ActiveRecord::Base.connected? will return true. But a call to ModelClass.find() or anything else that touches the database through the model class will raise an error.

This is because the @@connection_cache for this thread now contains an entry for ActiveRecord::Base pointing to a good connection, and one for ModelClass pointing to the now defunct connection we had originally.

We can fix this by calling ModelClass.reconnect! of course, which will allow us to connect to the database through ModelClass, but this is far from optimal:

- With many ActiveRecord subclasses we could potentially have multiple cases of defunct connections (the above is a simple scenario, there are worse cases) - We would now have at least 2 active connections to the database - We need to prefix pretty much every database accessing call with a check / force connection clause

Admittedly, this is unlikely to occur for rails apps; I hit this while working on a massively threaded transaction processing daemon, but it should really be fixed while we're here

I've modified the patch to include a fix for this.

Simon

02/09/06 08:15:40 changed by merul.patel@gmail.com

In addition to the points Simon has made above, there appears to be one other problem with the connection_specification.remove_connection method.

The issue is that the @@connection_cache still contains an empty hash corresponding to the thread's connection after the connection itself is disconnected.

I suggest two additional lines are added to clear out all expired Thread connections from the @@connection_cache.

def self.remove_connection(klass=self)
  spec = @@defined_connections[klass.name]
  konn = active_connections[klass.name]
  @@defined_connections.delete_if{|key, value| value == spec}
  @@connection_cache[Thread.current.object_id].delete_if{|key, value| value == konn}

# Delete all dead threads from @@connection_cache
  good_threads = Thread.list.map{|t| t.object_id }
  @@connection_cache.delete_if{|key,value| !good_threads.include?(key)} 

  active_connections.delete_if{|key, value| value == konn}
  konn.disconnect! if konn
  spec.config if spec
end

02/26/06 00:56:34 changed by david

  • owner changed from David to rails@bitsweat.net.

02/26/06 22:42:41 changed by bitsweat

  • owner changed from rails@bitsweat.net to bitsweat.
  • milestone set to 1.1.

02/26/06 23:12:06 changed by bitsweat

(In [3674]) ActiveRecord::Base.remove_connection explicitly closes database connections and doesn't corrupt the connection cache. Introducing the disconnect! instance method for the PostgreSQL, MySQL, and SQL Server adapters; implementations for the others are welcome. References #3591.

02/26/06 23:26:53 changed by bitsweat

  • status changed from new to closed.
  • resolution set to fixed.

(In [3676]) r3847@asus: jeremy | 2006-02-26 15:26:53 -0800

Apply [3674] to stable. Closes #3591.