RE: Large Array Clogging
RE: Large Array Clogging
- Subject: RE: Large Array Clogging
- From: "James C. Lee" <email@hidden>
- Date: Sun, 6 Jan 2008 14:55:37 -0800
- Importance: Normal
I should clarify that there are separate editing contexts for the parents
and children:
public void foo() {
NSArray parents;
NSArray children;
Enumeration anEnum;
EOEditingContext ecParent, ecChildren;
ecParent = new EOEditingContext();
// fetch parents with ecParent...
anEnum = parents.objectEnumerator();
while (anEnum.hasMoreElements()) {
aParent = (Parent)anEnum.nextElement();
ecChildren= new EOEditingContext();
children = fetch_with_editingContext(ecChildren);
// processing...
children = null;
ecChildren();
ecChildren= null;
System.gc();
}
}
^James
> -----Original Message-----
> From:
> webobjects-dev-bounces+jcl_applewodev=email@hidden
> ple.com
> [mailto:webobjects-dev-bounces+jcl_applewodev=dreamissary.com@
> lists.apple.com] On Behalf Of James C. Lee
> Sent: Sunday, January 06, 2008 2:46 PM
> To: 'WebObjects-Dev List'
> Subject: RE: Large Array Clogging
>
>
> Drew,
>
> Try this:
>
> public void foo() {
> NSArray parents;
> NSArray children;
> Enumeration anEnum;
> EOEditingContext editingContext;
>
> // fetch parents
> anEnum = parents.objectEnumerator();
> while (anEnum.hasMoreElements()) {
> aParent = (Parent)anEnum.nextElement();
> editingContext = new EOEditingContext();
> children = fetch_with_editingContext(editingContext);
> // processing...
> children = null;
> editingContext.dispose();
> editingContext = null;
> System.gc();
> }
> }
>
> ^James
>
>
> > -----Original Message-----
> > From:
> > webobjects-dev-bounces+jcl_applewodev=email@hidden
> > ple.com
> > [mailto:webobjects-dev-bounces+jcl_applewodev=dreamissary.com@
> > lists.apple.com] On Behalf Of Chuck Hill
> > Sent: Sunday, January 06, 2008 1:57 PM
> > To: Drew Thoeni
> > Cc: WebObjects-Dev List
> > Subject: Re: Large Array Clogging
> >
> >
> >
> > On Jan 6, 2008, at 11:53 AM, Drew Thoeni wrote:
> >
> > > I have two tables in a parent-child relationship. Some of the
> > > parents have a few children (say a few dozen) and others have many
> > > children (over 100,000). I'm trying to run through the
> parents and
> > > pre-calculate some statistics for their children (for example,
> > > average, mode, standard deviation, etc.).
> >
> > I'd be tempted to take a long, hard look at some of the aggregate
> > functions in Wonder that generate SQL for you. Batch
> processing is
> > not EOF's strongest area due to the overhead of object
> creation and
> > garbage collection.
> >
> >
> > > All this is working except when I reach a specific parent
> (#457) the
> > > processing just hangs. I ran this through debug and it stalls
> > > at the creation of the array for the children.
> > >
> > > However, the system would have already processed a similar size
> > > array (#30 has 118,000 children) and it seems it's in a loop.
> > > Activity monitor says the java process is taking 100%.
> > >
> > > Other background. I only save the ec after every 20
> updates to the
> > > parent records (to reduce write time). And I use
> > > ec.invalidateAllObjects() right after saving to clear our memory.
> >
> > Rather than do that, I'd create a new EC and just not retain _any_
> > references to the objects previously processed, the previous
> > EC etc.
> > EOF snapshot counting and Java GC should take care of the rest.
> >
> >
> > > Finally, there is nothing wrong with the data in
> parent/children of
> > > #457 as I can start the process there (or a few parents ahead of
> > > this) and it runs fine.
> > >
> > > This seems like a memory problem, but I don't get an out
> of memory
> > > exception.
> >
> > Before OutOfMemory, there is memory starvation. I think that
> > is what
> > is happening to you. Increase the heap size and see if
> that delays
> > the problem to a later parent. You could also launch with a JVM
> > parameter of -verbose:gc to log when GC happens.
> >
> > The solution is to clean up your handling of objects and editing
> > contexts.
> >
> > Chuck
> >
> > --
> >
> > Practical WebObjects - for developers who want to increase their
> > overall knowledge of WebObjects or who are trying to solve
> specific
> > problems.
> http://www.global-village.net/products/practical_webobjects
> >
> >
> >
> >
> >
> > _______________________________________________
> > Do not post admin requests to the list. They will be ignored.
> > Webobjects-dev mailing list (email@hidden)
> > Help/Unsubscribe/Update your Subscription:
> email@hidden
>
> This email sent to email@hidden
>
> _______________________________________________
> Do not post admin requests to the list. They will be ignored.
> Webobjects-dev mailing list (email@hidden)
> Help/Unsubscribe/Update your Subscription:
email@hidden
This email sent to email@hidden
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden