Re: Memory Management
Re: Memory Management
- Subject: Re: Memory Management
- From: Chuck Hill <email@hidden>
- Date: Fri, 6 Mar 2009 17:05:15 -0800
On Mar 6, 2009, at 4:57 PM, Jeff Schmitz wrote:
On Mar 6, 2009, at 6:19 PM, Chuck Hill wrote:
On Mar 4, 2009, at 6:30 PM, Jeff Schmitz wrote:
A little more info on my problem (yes I'm confused). For my
processing, all my data is not in one table. I need to work on
one coherent set of data at a time that is related across tables
via joins.
My tables are setup as follows:
Pool----*>>Entry-----63>>Game-----2>TeamPopup
To do my processing, specifically I need the rows
Tables? Rows? Joins? If you are thinking in SQL terms, one of
two things is probably true:
1. You should be doing this in SQL, not EOF
2. You are approaching it the wrong way
This is a transplanted OO design from when I didn't use EOF at all,
just flat files and RAM. Everything was read into ram at startup,
so fetching from the database wasn't a concern.
So, yes I need to do some re-architecting with EOF in mind, but this
is a phased process and is pretty much what I'm stuck with for now.
from the Entry table, the Game table and the TeamPopup table that
are associated with a single pool via joins from the pool's
entries to each entry's games to each game's teams.
To process 1 pool that has say 300 entries, each entry having 63
Games and each Game having 2 Teams, That's 63 rows from the Entry
table, 18900 distinct rows from the Games table and 37800 distinct
rows from the Teams table. My first question is, with a pool name
in hand, how do I fashion the qualifier so that all the Pool's
associated Game Rows and Team rows that I need for processing are
batch fetched in?
Have you turned on SQL logging to see what is getting sent to the
database?
No.
Without knowing what is actually happening, you have no (realistic)
hope of optimizing it. Turn it on. Look at what each like is
causing. Too many single row selects? Would an index improve
performance?
Does your model have Batch Faulting configured for these
relationships?
http://wiki.objectstyle.org/confluence/display/WO/EOF-Modeling-EOModeler#EOF-Modeling-EOModeler-BatchFaulting
Yes
Details?
Are you using prefetching keypaths with the fetch spec?
http://developer.apple.com/documentation/MacOSXServer/Reference/WO54_Reference/com/webobjects/eocontrol/EOFetchSpecification.html#setPrefetchingRelationshipKeyPaths(com.webobjects.foundation.NSArray)
I'm using Wonder, which I think does that. Hopefully I'm using it
correctly...
ERXBatchFetchUtilities.batchFetch(currPool.entries(),
Entry.GAMES.append(Game.TEAM_POPUPS));
That is _batch_ fetching, not _pre_fetching. They do similar things,
but prefetching is easier to use for most cases.
In the end, by creating a new EC and a new, dedicated Object Store
to process each pool, the performance isn't too bad and it doesn't
degrade, but sounds like the raw row route might be even faster, or
do the "joins" (I'm no SQL/DB guru) make that more difficult?
Faster than what? If what you have is not tuned now, fixing that
might make it faster and better. With raw rows you lose EOs. You get
raw pieces of data. Which brings us back to, "Maybe this is best
done in SQL".
Chuck
If it helps, I currently process the data something like this
(yes, my actual code is better structured, but this shows the
inner workings of what's really running).
Pool currPool = Pool.fetchStandardPoolWithName(ec, poolName);
ListIterator<Entry> entryIter = pool.entries().listIterator();
Entry nextEntry;
while (entryIter.hasNext()) {
nextEntry = entryIter.next();
for (int regionIndex = 0; regionIndex < 4; regionIndex ++) {
for (int gameIndex = 0; gameIndex < 15; j++) {
EOQualifier gameQual =
Game.GROUP.eq(regionIndex).and(Game.GAME.eq(gameIndex));
Game currentGame = nextEntry.games(gameQual).objectAtIndex(0);
EOQualifier selQual =
TeamPopup.COMBO_INDEX.eq(currentGame.selectedItem());
int seedOfSelectedTeam =
currentGame.teamPopups(selQual).lastObject().seed();
// Do stuff with the seed and retrieve other
information about the teams and games
}
}
}
So in the end, my first questions is regarding:
First, create a fetch specification for the main EO that you wish
to operate on and prep it for pulling out the primary key;
How do I create the fetch specification when I need to work on
multiple related EOs at the same time?
On Mar 1, 2009, at 10:04 PM, Andrew Lindesay wrote:
Hello Jeff;
Fair enough. I will keep it simple...
First, create a fetch specification for the main EO that you wish
to operate on and prep it for pulling out the primary key;
EOEntity fooE =
EOModelGroup.globalModelGroup().entityNamed("Foo");
EOFetchSpecification fs = ...
fs.setFetchesRawRows(true);
fs.setRawRowKeyPaths(fooE.primaryKeyAttributeNames());
By doing this, you don't give EOF so much work to do ensuring the
EO's you are working with are unique. OK so now fetch this into
an NSArray...
NSArray rrs = null;
{
EOEditingContext ec = new EOEditingContext();
ec.lock();
try { rrs = ec.objectsWithFetchSpecification(); }
finally { ec.unlock(); }
}
...work through these raw rows 50 (as an example) at a time...
for(int i=0;i<rrs.count();i+=50)
{
NSMutableArray<EOQualifier> qs = new
NSMutableArray<EOQualifier>();
for(int j=i;(j<(i+50)) && (j<rrs.count());j++)
{
EOGlobalID gid = fooE.globalIDForRow((NSDictionary)
rrs.objectAtIndex(j));
qs.addObject(LEEOHelper.qualifierForGlobalID(gid)); <--- copy
this method from LEWOStuff source.
}
EOEditingContext ec = new EOEditingContext();
ec.lock();
try
{
EOFetchSpecification fs = new EOFetchSpecification("Foo",new
EOOrQualifier(qs),null);
...fetch those EO's and do some stuff with them...
}
finally { ec.unlock(); }
}
...hope this helps and post back to the list (CC me) if you are
confused.
cheers.
Yes, I saw your post. However, I'm not an advanced EOF user,
nor do I know SQL very well (it's one of the reason I like
webobjects). That and the fact that any change I make needs to
be working in a couple of weeks makes me want to save a change
like you suggest for later when I've got more time to come up to
speed and to test. Maybe it's not as hard as it sounds? Keep
in mind that right now when you say "Load the list of EO's into
memory to be processed as raw-rows with the PK in the raw
rows." I have NO idea what you're talking about.
___
Andrew Lindesay
www.lindesay.co.nz
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden
--
Chuck Hill Senior Consultant / VP Development
Practical WebObjects - for developers who want to increase their
overall knowledge of WebObjects or who are trying to solve specific
problems.
http://www.global-village.net/products/practical_webobjects
--
Chuck Hill Senior Consultant / VP Development
Practical WebObjects - for developers who want to increase their
overall knowledge of WebObjects or who are trying to solve specific
problems.
http://www.global-village.net/products/practical_webobjects
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden