Expanding Import
Expanding Import
- Subject: Expanding Import
- From: Scott Winn <email@hidden>
- Date: Tue, 7 Mar 2006 13:49:15 -0800
Hello List,
I am working on a fixed-width record importer for my database. I
have to interface with a legacy app, so there is no getting out of it.
My WO importer app has significant performance issues. The CPU is
very busy, memory usage is high, and the database has hardly any load
on it. I have increased my JVM size, I ec.dispose() and get a new EC
periodically, and I run garbage collection after every imported
file. I don't get out of memory errors anymore, but after importing
a few months worth of data the app runs so slow that it might as well
be stopped.
As I read the files, I get codes for an Entity like NCStation, for
example. Then I check for a match on stationCode in the database
using a fetchSpec, create it if it doesn't exist, and add it to the
appropriate relationship to a Certificate in this case. To see what
is going on, I called ec.updatedObjects() just before an
ec.saveChanges() and I got something like this. . .
values = {
stationCode = "199";
stationDescription = <com.webobjects.foundation.NSKeyValueCoding$Null>;
Certificates = (
--> "<NCCertificate b94a3f _EOIntegralKeyGlobalID[Certificate
(java.lang.Integer)1]>",
--> "<NCCertificate b9ae95 _EOIntegralKeyGlobalID[Certificate
(java.lang.Integer)2]>",
--> "<NCCertificate 74c043 _EOIntegralKeyGlobalID[Certificate
(java.lang.Integer)3]>",
--> "<NCCertificate faa3c5 _EOIntegralKeyGlobalID[Certificate
(java.lang.Integer)4]>",
--> "<NCCertificate 15ebf0 _EOIntegralKeyGlobalID[Certificate
(java.lang.Integer)5]>",
"<NCCertificate 75deb1 <EOTemporaryGlobalID: 0 0 -64 -88 42 -3 0 0
-60 50 22 0 0 0 1 9 -73 45 -46 -91 86 -26 -122 58>>");
Company = "<NCCompany fc9484 _EOIntegralKeyGlobalID[Company
(java.lang.Integer)1]>";
stationName = <com.webobjects.foundation.NSKeyValueCoding$Null>;
};
this = "<NCStation 9096c6 _EOIntegralKeyGlobalID[Station
(java.lang.Integer)1]>";
The lines indicated with --> seem to just be along for the ride. I
don't need to make any changes to them. I don't want to make any
changes to the entity itself. All I want to do is add more objects
to the relationship. After two or three imported files, my ec.
updatedObjects() call is showing thousands of entries for the "to-
many" relationships. Obviously, it just gets worse the more I import.
So my questions are these. . .
1) Is my diagnosis on the right track?
2) What is the right way to approach something like this? Raw Rows?
Or would that make any difference?
And the big question. . .
3) How do I check the database for an Entity and relate an object to
it when there are potentially a million other objects related to it?
Any help is sincerely appreciated,
Scott
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden