Re: Manipulating large amounts of data with EOF - feedback, anyone?
Re: Manipulating large amounts of data with EOF - feedback, anyone?
- Subject: Re: Manipulating large amounts of data with EOF - feedback, anyone?
- From: Ray Kiddy <email@hidden>
- Date: Fri, 9 Jan 2009 15:39:32 -0800
On Jan 9, 2009, at 1:58 PM, Hugi Thordarson wrote:
Good evening folks!
The databases I'm responsible for contain a lot of data and I find
myself frequently needing to resort to boring stuff like raw row
fetching to create large reports or otherwise handle a lot of data.
But sometimes, even that isn't enough - an array of ten million
items is difficult for any application to handle, even though the
ten million objects are just NSDictionaries/raw rows. Besides -
working with raw rows is no fun. I'm spoiled by years of EOF-y
goodness.
<snip>
Anyway, I would love to hear how other folks are handling huge
datasets. I would love fedback on the technique I'm using, and ieas
for improvement would be great. Just about the only idea I'm not
open to is "just use JDBC" ;-). I've been there and I don't want to
be there. That's why I'm using EOF :-).
Cheers,
- Hugi
I end up doing things in SQL. I find that, more often than I sometimes
like, I have a suite of perl scripts which do some fetches, generate
SQL and pipe that back into my database. EOF is just not comfortable
handling large data sets. This is especially true when importing data.
The iTunes store handles large amounts of data and vend it quickly but
they go through herculean efforts and have a bunch of the original WO
developers to help them, so they do not really count.
There are some things EOF could do to help people use large data sets,
and perhaps these could be done in Project Wonder instead.
- support SQL cursors
- have some support for partial fetches
- have better support for indexes and help make sure they get used
And I am sure there is some other stuff I am not remembering at the
moment....
Basically, EOF could be flexible enough to take advantage of the
smartest thing that the database being used at any particular time
could do. How EOF has traditionally worked is that it takes the lowest
common denominator of what might be available and, really, we are
talking about the lowest common denominator at about the time Oracle 7
came out. It could definitely do better.
On the other hand, it is arguable that EOF was designed for agile
model design, for being easier to develop with, and not for the kind
of brute-strength large data set manipulation that we are discussing.
I have thought about trying to get EOF to do some of this, but when I
am doing some dumb-simple SQL-ish thing, trying to use EOF seems like
overkill.
- ray
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden