RE: Large Array Clogging
RE: Large Array Clogging
- Subject: RE: Large Array Clogging
- From: "James C. Lee" <email@hidden>
- Date: Wed, 9 Jan 2008 21:05:49 -0800
- Importance: Normal
Thought I'd follow up just in case someone needs this in the future and
searches the archive. Here's what we did (the code in my original email
missed the dispose()). Thanks for the reminder, Pierre!
public void foo() {
NSArray parents, children;
Enumeration anEnum;
EOEditingContext ecParents, ecChildren;
while (fetch more parents) {
ecParents = new EOEditingContext();
// fetch parents with ecParents in 1000 or n increments...
anEnum = parents.objectEnumerator();
while (anEnum.hasMoreElements()) {
aParent = (Parent)anEnum.nextElement();
ecChildren= new EOEditingContext();
children = fetch_with_editingContext(ecChildren);
// processing...
children = null;
ecChildren.dispose();
ecChildren= null;
System.gc();
}
parents = null;
ecParents.dispose();
ecParents = null;
System.gc();
}
}
^James
> -----Original Message-----
> From: Mr. Pierre Frisch [mailto:email@hidden]
> Sent: Wednesday, January 09, 2008 3:44 PM
> To: Drew Thoeni
> Cc: Mr. Pierre Frisch; James C. Lee; WebObjects-Dev List
> Subject: Re: Large Array Clogging
>
>
> May I suggest that you call dispose() on the EC when
> discarding it. It
> will accelerate the GC as this will actively break the object
> relations.
>
> Pierre
> --
> Pierre Frisch
> email@hidden
>
>
> On Jan 9, 2008, at 15:04, Drew Thoeni wrote:
>
> > James,
> >
> > Implemented this yesterday and it works as advertised, that is,
> > perfectly. I can calculate all the stats for the 18,000
> parents and
> > 100,000,000 children in an hour too! Thanks for the pattern.
> >
> > Best,
> >
> > Drew
> >
> > On Jan 6, 2008, at 5:55 PM, James C. Lee wrote:
> >
> >> I should clarify that there are separate editing contexts for the
> >> parents
> >> and children:
> >>
> >> public void foo() {
> >> NSArray parents;
> >> NSArray children;
> >> Enumeration anEnum;
> >> EOEditingContext ecParent, ecChildren;
> >>
> >> ecParent = new EOEditingContext();
> >> // fetch parents with ecParent...
> >> anEnum = parents.objectEnumerator();
> >> while (anEnum.hasMoreElements()) {
> >> aParent = (Parent)anEnum.nextElement();
> >> ecChildren= new EOEditingContext();
> >> children = fetch_with_editingContext(ecChildren);
> >> // processing...
> >> children = null;
> >> ecChildren();
> >> ecChildren= null;
> >> System.gc();
> >> }
> >> }
> >>
> >> ^James
> >>
> >>
> >>> -----Original Message-----
> >>> From:
> >>> webobjects-dev-bounces+jcl_applewodev=email@hidden
> >>> ple.com
> >>> [mailto:webobjects-dev-bounces+jcl_applewodev=dreamissary.com@
> >>> lists.apple.com] On Behalf Of James C. Lee
> >>> Sent: Sunday, January 06, 2008 2:46 PM
> >>> To: 'WebObjects-Dev List'
> >>> Subject: RE: Large Array Clogging
> >>>
> >>>
> >>> Drew,
> >>>
> >>> Try this:
> >>>
> >>> public void foo() {
> >>> NSArray parents;
> >>> NSArray children;
> >>> Enumeration anEnum;
> >>> EOEditingContext editingContext;
> >>>
> >>> // fetch parents
> >>> anEnum = parents.objectEnumerator();
> >>> while (anEnum.hasMoreElements()) {
> >>> aParent = (Parent)anEnum.nextElement();
> >>> editingContext = new EOEditingContext();
> >>> children = fetch_with_editingContext(editingContext);
> >>> // processing...
> >>> children = null;
> >>> editingContext.dispose();
> >>> editingContext = null;
> >>> System.gc();
> >>> }
> >>> }
> >>>
> >>> ^James
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From:
> >>>> webobjects-dev-bounces+jcl_applewodev=email@hidden
> >>>> ple.com
> >>>> [mailto:webobjects-dev-bounces+jcl_applewodev=dreamissary.com@
> >>>> lists.apple.com] On Behalf Of Chuck Hill
> >>>> Sent: Sunday, January 06, 2008 1:57 PM
> >>>> To: Drew Thoeni
> >>>> Cc: WebObjects-Dev List
> >>>> Subject: Re: Large Array Clogging
> >>>>
> >>>>
> >>>>
> >>>> On Jan 6, 2008, at 11:53 AM, Drew Thoeni wrote:
> >>>>
> >>>>> I have two tables in a parent-child relationship. Some of the
> >>>>> parents have a few children (say a few dozen) and
> others have many
> >>>>> children (over 100,000). I'm trying to run through the
> >>> parents and
> >>>>> pre-calculate some statistics for their children (for example,
> >>>>> average, mode, standard deviation, etc.).
> >>>>
> >>>> I'd be tempted to take a long, hard look at some of the
> aggregate
> >>>> functions in Wonder that generate SQL for you. Batch
> >>> processing is
> >>>> not EOF's strongest area due to the overhead of object
> >>> creation and
> >>>> garbage collection.
> >>>>
> >>>>
> >>>>> All this is working except when I reach a specific parent
> >>> (#457) the
> >>>>> processing just hangs. I ran this through debug and it
> stalls at
> >>>>> the creation of the array for the children.
> >>>>>
> >>>>> However, the system would have already processed a similar size
> >>>>> array (#30 has 118,000 children) and it seems it's in a loop.
> >>>>> Activity monitor says the java process is taking 100%.
> >>>>>
> >>>>> Other background. I only save the ec after every 20
> >>> updates to the
> >>>>> parent records (to reduce write time). And I use
> >>>>> ec.invalidateAllObjects() right after saving to clear
> our memory.
> >>>>
> >>>> Rather than do that, I'd create a new EC and just not
> retain _any_
> >>>> references to the objects previously processed, the previous EC
> >>>> etc. EOF snapshot counting and Java GC should take care of the
> >>>> rest.
> >>>>
> >>>>
> >>>>> Finally, there is nothing wrong with the data in
> >>> parent/children of
> >>>>> #457 as I can start the process there (or a few parents ahead of
> >>>>> this) and it runs fine.
> >>>>>
> >>>>> This seems like a memory problem, but I don't get an out
> >>> of memory
> >>>>> exception.
> >>>>
> >>>> Before OutOfMemory, there is memory starvation. I think that is
> >>>> what is happening to you. Increase the heap size and see if
> >>> that delays
> >>>> the problem to a later parent. You could also launch with a JVM
> >>>> parameter of -verbose:gc to log when GC happens.
> >>>>
> >>>> The solution is to clean up your handling of objects and editing
> >>>> contexts.
> >>>>
> >>>> Chuck
> >>>>
> >>>> --
> >>>>
> >>>> Practical WebObjects - for developers who want to increase their
> >>>> overall knowledge of WebObjects or who are trying to solve
> >>> specific
> >>>> problems.
> >>> http://www.global-village.net/products/practical_webobjects
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Do not post admin requests to the list. They will be ignored.
> >>>> Webobjects-dev mailing list (email@hidden)
> >>>> Help/Unsubscribe/Update your Subscription:
> >>> email@hidden
> >>>
> >>> This email sent to email@hidden
> >>>
> >>> _______________________________________________
> >>> Do not post admin requests to the list. They will be ignored.
> >>> Webobjects-dev mailing list (email@hidden)
> >>> Help/Unsubscribe/Update your Subscription:
> >> email@hidden
> >>
> >> This email sent to email@hidden
> >>
> >> _______________________________________________
> >> Do not post admin requests to the list. They will be ignored.
> >> Webobjects-dev mailing list (email@hidden)
> >> Help/Unsubscribe/Update your Subscription:
> >>
> >> t.net
> >>
> >> This email sent to email@hidden
> >
> > _______________________________________________
> > Do not post admin requests to the list. They will be ignored.
> > Webobjects-dev mailing list (email@hidden)
> > Help/Unsubscribe/Update your Subscription:
> >
> > om
> >
> > This email sent to email@hidden
>
>
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden