Ruby on Rails | Screencasts | Download | Documentation | Weblog | Community | Source

Ticket #738 (closed enhancement: incomplete)

Opened 4 years ago

Last modified 2 years ago

[PATCH] ActiveRecord should provide an '''each''' { ''block'' } method

Reported by: Skolem Assigned to: bitsweat
Priority: normal Milestone: 2.x
Component: ActiveRecord Version: edge
Severity: normal Keywords: each enumerable
Cc: ivey, wonko9

Description

ActiveRecord should provide a method each(conditions=nil,orderings=nil,limit=nil,joins=nil,&block). each() passes every row returned by the select to the block, without slurping all the records into memory at once. This allows one to process each record of a very large table without using excessive memory. This could be implemented by simply opening the select query and simply passing each row to the block, without building up a list of rows in memory.

Attachments

find_with_block.diff (7.7 kB) - added by jack on 04/12/07 14:55:49.
find_with_block
find_each.rb (8.3 kB) - added by jack on 05/23/07 08:25:44.
New patch, where the block functionality is moved to seperate each method. So now Model.each { block } for new block feature or Model.find() for old find.
find_with_block.2.diff (7.1 kB) - added by josh on 06/25/07 20:33:48.
mysql_activerecord_streams.rb (3.2 kB) - added by wonko9 on 07/17/07 20:41:55.
activerecord streams monkeypatch

Change History

(follow-up: ↓ 9 ) 03/02/05 00:45:44 changed by nzkoz

  • milestone set to 1.x.

I'm setting the milestone of this ticket to 1.x, this means it will be implemented after 1.0. If you feel this functionality should be included in the 1.0 release please email the developers list to discuss it.

07/07/06 21:51:28 changed by bitsweat

  • status changed from new to closed.
  • resolution set to fixed.
  • milestone changed from 1.1 to 1.x.

Please reopen with a patch and test cases if you pursue the feature.

04/12/07 14:55:49 changed by jack

  • attachment find_with_block.diff added.

find_with_block

04/12/07 14:56:52 changed by jack

  • status changed from closed to reopened.
  • resolution deleted.

This feature would be very useful as it dramatically lowers memory usage with large tables - i attach a dirty patch that seems to add a basic working block usage (for mysql adapter) like Model.find(:all, :include=>:other_model) {|m| puts m.name} . It would be great if some rails developer would make a real feature like that in next release.

04/12/07 18:38:15 changed by ivey

  • cc set to ivey.

05/22/07 19:43:01 changed by josh

  • summary changed from ActiveRecord should provide an '''each''' { ''block'' } method to [PATCH] ActiveRecord should provide an '''each''' { ''block'' } method.

05/22/07 21:28:12 changed by bitsweat

  • keywords set to each enumerable.
  • owner changed from David to bitsweat.
  • status changed from reopened to new.

05/23/07 08:25:44 changed by jack

  • attachment find_each.rb added.

New patch, where the block functionality is moved to seperate each method. So now Model.each { block } for new block feature or Model.find() for old find.

06/25/07 19:38:51 changed by josh

Jamis also has a related post on this, Faking cursors in ActiveRecord.

06/25/07 20:32:53 changed by josh

  • version changed from 0.10.0 to edge.

06/25/07 20:33:48 changed by josh

  • attachment find_with_block.2.diff added.

07/17/07 20:41:55 changed by wonko9

  • attachment mysql_activerecord_streams.rb added.

activerecord streams monkeypatch

(in reply to: ↑ 1 ) 07/17/07 20:44:30 changed by wonko9

  • cc changed from ivey to ivey, wonko9.

I created a monkeypatch for your patch and did some testing with it.

We have a table with over 6MM very large rows. As a test I did a simple TableClass.find(:all) {next} and watched as the memory went up and up without letting go. Did I miss something in the monkeypatch or is there a larger issues with AR not taking this streaming well?

10/15/07 01:23:18 changed by bitsweat

This is because it doesn't matter whether you yield rows one by one, MySQL still pulls all the query results at once.

This patch is nice for aesthetics but will only really work as advertised if it uses paging or cursors under the covers.

10/15/07 02:22:59 changed by bitsweat

  • status changed from new to closed.
  • resolution set to incomplete.