Re: I'm killing my database
Re: I'm killing my database
- Subject: Re: I'm killing my database
- From: Scott Winn <email@hidden>
- Date: Thu, 26 Apr 2007 08:19:09 -0700
When I have put a lot of data through the Importer previously I had
the application crash with a SIGBUS error. On a separate occasion,
the app didn't crash, but when I went to log out I got "Internal
Server Error has occurred". Looking at the run log showed three null
pointer exceptions encountered; trying to dispose the editing context
from the look of them (I should have saved those exception
messages). Both of these problams happened in previous tests before
the DB crash.
I'm sure both of those incidents should raise warning flags, but I'm
not sure what those flags mean. Do they point to the same problem?
I found basically no information on SIGBUS errors. I can run
everything through from the top and see if I have the same issues,
but I think I'll try to get performance boosted a bit first. I will
look into batch faulting and disposing the editing context at regular
intervals.
I should mention that all of this is happening on a WOLongResponse
page, so a user can watch the import progress and fix any errors in
the source files.
Sincere appreciation for all the help,
Scott
On Apr 25, 2007, at 5:44 PM, Chuck Hill wrote:
On Apr 25, 2007, at 12:03 PM, Scott Winn wrote:
Thanks Chuck. I will give it a look.
I still don't know what the problem was, but I have updated
FrontBase and its plugin (I was a few point releases behind). The
release notes for some of those points included delete fixes. I
also added EC.undoManager().removeAllActions(). I haven't
disposed of the editing context yet, but I may once I have checked
the performance results from my most recent run.
I put the same data back through the importer and no DB crash this
time,
That is from the FB update.
although things still seemed very slow. More than 1/2 an hour to
process about 1 MB of data seem like a bit much. I checked free
memory just prior to doing the delete and it never dips below 74%.
I have no doubt that it can be optimized more than that. Make sure
you have batch faulting configured in the model.
Chuck
On Apr 24, 2007, at 10:32 PM, Guido Neitzer wrote:
On 24.04.2007, at 20:59, Chuck Hill wrote:
The first thing I would do is to add this line:
recordContext .undoManager().removeAllActions();
The next thing I would do (voodoo or not) is to periodically
dispose of recordContext and create a replacement. In the past
that was vital.
It still is. It helps. Not necessarily for performance but
normally for memory usage. I normally do bulk import in batches
of up to 500 objects per batch, then save and dispose the
editingContext, set it to null and create in new one in the next
batch. Also for bulk updating of EOs.
That works well for my importers.
If you need to resolve relationships during import, pre-fetching
the relationships can make a huge difference.
As as you are deleting, I have a DeletePrefetcher class that may
be of use. I originally wrote this as a bug workaround, but it
might also make the fetches more efficient. It is part of our
GVCEOFExtensions framework that you can get from http://
sourceforge.net/projects/gvcsitemaker
To save you the download, I have temporarily stuck a copy here:
http://www.gvcsitemaker.com/chill/files/deleteprefetcher.java
Chuck
--
Practical WebObjects - for developers who want to increase their
overall knowledge of WebObjects or who are trying to solve
specific problems.
http://www.global-village.net/products/practical_webobjects
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
40global-village.net
This email sent to email@hidden
--
Practical WebObjects - for developers who want to increase their
overall knowledge of WebObjects or who are trying to solve specific
problems.
http://www.global-village.net/products/practical_webobjects
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden