Re: Out of memory
Re: Out of memory
- Subject: Re: Out of memory
- From: Kieran Kelleher <email@hidden>
- Date: Wed, 7 May 2008 12:56:50 -0400
Thanks Miguel .... I had not used jconsole before.... nice, simple and
free.
I have some background processes that risk running out of memory such
as importing millions of rows of data from CSV files and creating
EO's. I know that EC recycling and many other strategies are
recommended and for those processes I use such strategies, however, I
have found that the use of the attached class saves my app from dying
if it gets close to out of memory. All it does is check the memory
usage and if too close for comfort it just forces garbage collection.
Often this will push memory way down. This is probably of limited use
where you have a big EOF operation inflating losts of EO's. If you
memory problem is a gazillion sessions logged into the app, it may not
help much at all.
It was inspired by some code I saw in Wonder at one stage. It was born
out of desperation trying to get files with 10+ million rows to import
and the usual strategies were not working and I really did not want to
insert these as raw rows directly using adaptor operations since I
wanted the full EOF validation features to skip non-conforming import
rows.
Usage:
call the static method forceGarbageCollectionIfNecessary
Configuration using property. Less than 1.0 is a percentage memory
usage threshold. Great than 1.0 is assumed a Megabyte free minimum
threshold. Like I said look at Wonder code for a similar calculation
somewhere. IIRC the difference between this and Wonder is that I put
the run finalization inside the gc repeating loop.
YMMV,
Hope it is of use, Kieran
package wk.foundation;
import java.math.BigDecimal;
import org.apache.log4j.Logger;
public class WKRuntimeUtilities {
private static final Logger log = Logger
.getLogger(WKRuntimeUtilities.class);
/**
*@returntrueiffreememoryislessthanthethreshold
*/
public static boolean isMemoryStarved() {
String thresholdString =
System.getProperty("wk.foundation.WKRuntimeUtilities.memoryThreshold",
"0.9");
BigDecimal MEMORY_THRESHOLD = new BigDecimal(thresholdString);
long max = Runtime.getRuntime().maxMemory();
long total = Runtime.getRuntime().totalMemory();
long free = Runtime.getRuntime().freeMemory() + (max - total);
long used = max - free;
long threshold = (long) (MEMORY_THRESHOLD.doubleValue() < 1.0 ?
MEMORY_THRESHOLD.doubleValue() * max : (max -
(MEMORY_THRESHOLD.doubleValue() * 1024 * 1024)));
boolean result = (used > threshold);
if (log.isDebugEnabled())
log.debug("max = " + max + "; total = " + total + "; free = " +
free + "; used = " + used + "; threshold = " + threshold
+ "; isMemoryStarved = " + result);
return result;
}
public static String memoryStatistics() {
long max = Runtime.getRuntime().maxMemory();
long total = Runtime.getRuntime().totalMemory();
long free = Runtime.getRuntime().freeMemory() + (max - total);
long used = max - free;
StringBuilder s = new StringBuilder("Memory Statistics");
s.append("\nMax = " +
WKFormatUtilities.userFriendlyByteCountString(max));
s.append("\nTotal = " +
WKFormatUtilities.userFriendlyByteCountString(total));
s.append("\nFree = " +
WKFormatUtilities.userFriendlyByteCountString(free));
s.append("\nUsed = " +
WKFormatUtilities.userFriendlyByteCountString(used));
s.append("\nUsed Percent = " + used * 100 / max + " %");
return s.toString();
}
/**
*Checksifmemoryisstarvedandthenforcesfullgarbagecollection
*/
public static void forceGarbageCollectionIfNecessary() {
try {
if (WKRuntimeUtilities.isMemoryStarved()) {
StringBuilder s = new StringBuilder("Notification of Forced
Garbage Collection");
s.append("\nBEFORE Forced GC = " +
WKRuntimeUtilities.memoryStatistics());
WKRuntimeUtilities.forceGC(0);
s.append("\nAFTER Forced GC = " +
WKRuntimeUtilities.memoryStatistics());
// For now log.error so I get emails, but when satisfied change to
log.warn
log.error("NOT an ERROR - Just Notification so Kieran is
aware of it.\n"
+ s.toString(), new Throwable());
}
} catch (Exception e) {
log.error("Error performing forced garbage collection. Check the
code", e);
}
}
/**
* Forces the garbage collector to run. The
* max loop parameter determines the maximum
* number of times to run the garbage collector
* if the memory footprint is still going down.
* In normal cases you would just need to call
* this method with the parameter 1. If called
* with the parameter 0 the garbage collector
* will continue to run until no more free memory
* is available to collect. <br/>
* <br/>
* Note: This can be a very costly operation and
* should only be used in extreme circumstances.
* @param maxLoop maximum times to run the garbage
* collector. Passing in 0 will cause the
* collector to run until all free objects
* have been collected.
*/
public static void forceGC(int maxLoop) {
log.warn("Forcing full Garbage Collection", new Throwable());
Runtime runtime = Runtime.getRuntime();
long isFree = runtime.freeMemory();
long wasFree;
int i=0;
do {
wasFree = isFree;
runtime.gc();
runtime.runFinalization();
isFree = runtime.freeMemory();
i++;
} while (isFree > wasFree && (maxLoop<=0 || i<maxLoop) );
}
}
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden