Re: Report generation: How to avoid "java.lang.OutOfMemoryError: Java heap space"?
Re: Report generation: How to avoid "java.lang.OutOfMemoryError: Java heap space"?
- Subject: Re: Report generation: How to avoid "java.lang.OutOfMemoryError: Java heap space"?
- From: Fabian Peters <email@hidden>
- Date: Fri, 28 Sep 2007 17:46:50 +0000
Hi all,
I tracked the problem down to a method in an EO that caused several
new temporary EO instances to be created in nested ECs. Once I
changed the code to use a single instance of the EO (still in a
nested EC) everything works as expected. I'm not really sure why this
could cause an OOM error though? (The temporary EO is modified to
conform to a particular standard and then used to compute a few
values, without ever getting persisted.)
To wrap things up: After the above change there's no significant
difference in memory use between a StringBuilder-based implementation
and using a WOComponent - YMMV. Batching is of course still
necessary, but now memory is being freed up as expected.
Thanks for all your suggestions,
Fabian
Am 28.09.2007 um 16:16 schrieb Simon McLean:
JProfiler works great with eclipse and wo apps.
I think you'll find components fall over because they have to hall
the entire array of results into memory. Not good if you have a
couple thousand results. The batch approach means you only hold one
batch of EOs in memory at a time (thanks to Guido from me too for
the pointer to ERXFetchSpecificationBatchiterator !!)
If you want to use components for rendering you might want to try
using the batch approach to load dictionaries or string arrays, and
build your components based on them. It won't be as scalable as
outputting to a file, but it might get you where you need to be.
Simon
On 28 Sep 2007, at 13:46, Fabian Peters wrote:
Hi Simon,
thanks for your response - that's about what I feared would be the
way to go.
Did anybody have a closer look at what makes components choke on
this? I tried profiling with Shark to shed some light on this, but
even if I stop the report generation to prevent the OOM error from
occurring, Shark fails to complete the sample collection. Is
JProfiler the only option for profiling with eclipse/wolips right
now?
Fabian
P.S.: Guido, thanks for the pointer to
ERXFetchSpecificationBatchIterator!
Am 27.09.2007 um 21:46 schrieb Simon McLean:
HI -
We are in a similar situation right now, trying to figure out the
best way of getting huge reports out of EOF whilst maintaining
the richness provided by Enterprise Objects.
Here's the first thing we figured: forget components. As you
appear to have found out they simply don't scale when you get
into thousands of rows.
At the moment we are playing with fetching just the pk's of the
objects required for the report (using raw rows), then cycling
through the array instantiating each object one by one (using a
temporary EC for each object - creating it to fetch the object,
doing it's stuff, then disposing of it) and appending a string of
the required attributes of each object to an XML document which
is ultimately returned by the method. We are then going to use a
reporting tool (Jasper, Pentaho, BIRT ?) to generate the reports
based on the XML document we have got from EOF.
Simon
On 27 Sep 2007, at 22:07, Fabian Peters wrote:
Hi all,
I'm trying to set up a simple reporting component, output of
which is to be used in a statistical analysis. It's basically a
WORepetition wrapped around about 40 WOStrings, supposed to
render a long list, CSV-formatted in this case. The WOStrings
are bound to various keypathes of "item" EO. Some of the
keypathes touch on relationships, so there are other objects
getting fetched as needed (about 7 per "item" EO on average).
What I did so far was to wrap the main repetition in a second
repetition that in turn is bound to a "batches" array, which
simply contains an index for the number of batches (e.g. "(1, 2,
3, 4, 5, 6)")
<batchingRepetition>
<mainRepetition>list contents</mainRepetition>
</batchingRepetition>
I then fetch only the objects for the current batch, using a
fresh EC for each batch (disposing the old one). I've used an
ERXBatchingDisplayGroup and then an
ERXEOControlUtilities.objectsInRange() for the batching, both
with the same results memory-wise. An example of the apps memory
usage is at the end of this mail. From what I see, the disposal
of the "old" EC (before the "lower range limit" log statement)
does not free any memory?
Increasing Java heap space is not really a solution, since this
will have to scale. Using SQL or JDBC also is not an option (or
at least one that I really want to avoid), since I'm depending
on code in the EOs.
Since I'm a bit lost right now, I hope someone can push me in
the right direction...
cheers,
Fabian
Memory usage ("registered objects" shows the EC's registered
objects count before disposal):
INFO 10,01 MB used/2,78 MB free com.myapp.WeightAnalysisPage:102
- list batches array: (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12,
13)
INFO 10,16 MB used/2,63 MB free com.myapp.WeightAnalysisPage:75
- registered objects: 0
INFO 10,25 MB used/2,54 MB free com.myapp.WeightAnalysisPage:80
- lower range limit is 0
INFO 10,31 MB used/2,48 MB free com.myapp.WeightAnalysisPage:82
- upper range limit is 100
INFO 20,79 MB used/5,97 MB free com.myapp.WeightAnalysisPage:75
- registered objects: 726
INFO 20,87 MB used/5,9 MB free com.myapp.WeightAnalysisPage:80 -
lower range limit is 100
INFO 20,94 MB used/5,83 MB free com.myapp.WeightAnalysisPage:82
- upper range limit is 200
INFO 29,59 MB used/11,16 MB free com.myapp.WeightAnalysisPage:75
- registered objects: 678
INFO 29,7 MB used/11,04 MB free com.myapp.WeightAnalysisPage:80
- lower range limit is 200
INFO 29,76 MB used/10,99 MB free com.myapp.WeightAnalysisPage:82
- upper range limit is 300
INFO 37,45 MB used/24,54 MB free com.myapp.WeightAnalysisPage:75
- registered objects: 716
INFO 37,54 MB used/24,46 MB free com.myapp.WeightAnalysisPage:80
- lower range limit is 300
INFO 37,62 MB used/24,38 MB free com.myapp.WeightAnalysisPage:82
- upper range limit is 400
INFO 49,25 MB used/12,75 MB free com.myapp.WeightAnalysisPage:75
- registered objects: 677
INFO 49,35 MB used/12,65 MB free com.myapp.WeightAnalysisPage:80
- lower range limit is 400
INFO 49,43 MB used/12,57 MB free com.myapp.WeightAnalysisPage:82
- upper range limit is 500
INFO 53,78 MB used/9,78 MB free com.myapp.WeightAnalysisPage:75
- registered objects: 680
INFO 53,92 MB used/9,64 MB free com.myapp.WeightAnalysisPage:80
- lower range limit is 500
INFO 54 MB used/9,56 MB free com.myapp.WeightAnalysisPage:82 -
upper range limit is 600
INFO 57,8 MB used/5,76 MB free com.myapp.WeightAnalysisPage:75 -
registered objects: 729
INFO 57,9 MB used/5,66 MB free com.myapp.WeightAnalysisPage:80 -
lower range limit is 600
INFO 57,91 MB used/5,65 MB free com.myapp.WeightAnalysisPage:82
- upper range limit is 700
INFO 63,12 MB used/450,52 KB free com.myapp.WeightAnalysisPage:
75 - registered objects: 678
INFO 63,14 MB used/435,39 KB free com.myapp.WeightAnalysisPage:
80 - lower range limit is 700
INFO 63,21 MB used/364 KB free com.myapp.WeightAnalysisPage:82 -
upper range limit is 800
WARN 63,28 MB used/291,5 KB free NSLog :41 -
<er.extensions.ERXDirectActionRequestHandler>: Exception while
handling action named "pages" on action class
"null" :com.webobjects.foundation.NSForwardException
[java.lang.OutOfMemoryError] Java heap space
[2007-09-27 20:51:34 GMT] <WorkerThread0>
com.webobjects.foundation.NSForwardException for
java.lang.OutOfMemoryError: Java heap space
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
email@hidden
This email sent to email@hidden
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden