Re: PrimaryKey generation takes a lot of time
Re: PrimaryKey generation takes a lot of time
- Subject: Re: PrimaryKey generation takes a lot of time
- From: Patrick Middleton <email@hidden>
- Date: Tue, 06 May 2014 09:40:55 +0100
We encountered this recently against MySQL 5.6.12.
(Not using Project Wonder)
The default primary key generation was invoking a stored procedure
once per row. Once per second, some sort of garbage collection would
occur inside the RDBMS and the stored procedure would take about 0.2s
to complete instead of about 0.005s.
The fix is to create and cache all your new primary keys at once in
advance using something like
er.extensions.jdbc.ERXJDBCAdaptor.Channel.primaryKeysForNewRowsWithEntit
y(int cnt, com.webobjects.eoaccess.EOEntity entity)
then use something like
databaseContextNewPrimaryKey(EODatabaseContext dbCtxt, Object object,
EOEntity entity)
http://wocommunity.org/documents/javadoc/WebObjects/5.4.2/com/
webobjects/eoaccess/EODatabaseContext.Delegate.html?is-external=true
to fetch a candidate new key from your cache rather than make a
database round-trip to make one.
On 6 May 2014, at 08:20, Tobias Janz wrote:
I forgott to replay to the list.
the database is close to the server. I did not find the way to copy
the Database from Mac OS Frontbase to CentOS Frontbase or the other
way. I tryed to recover me lokal Database with a Backup created
with CentOS-Frontbase, but it fails.
So I think the best way for me will be the WRITE ALL export.
I will take a look at this.
Thanks
Tobias
2014-05-06 7:58 GMT+02:00 Chuck Hill <email@hidden>:
Hi Tobias,
I missed the "Frontbase in the cloud on centOS” before. Is the app
server not “close” to the database server? With latency like this,
you are going to face production issues, too. That would be my
first concern.
Some ideas to consider:
Import it locally, do a WRITE ALL export and import on the server
Change to the binary UUID primary keys that are generated in the
application
ERXLongPrimaryKeyFactory
Chuck
On 2014-05-05, 10:28 PM, "Tobias Janz" wrote:
Hi Chuck,
thanks for your advice, but this approch i have allready tryed. I
create a new editing context for each 100. Also i've tryed multiple
threads with relevant success.
When i do the import on my lokal database its much much faster. I
think it's because of the latency. Is there a way to savely create
primary key with multiple DB-connections in multiple Threads? And
how do i get multiple database connections?
regards Tobias
2014-05-06 1:32 GMT+02:00 Chuck Hill <email@hidden>:
Are you creating the objects and saving them all at once? If so,
that is your problem. Save them in batches of 500 or 1000 and it
will go much quicker. You might want to try making a new editing
context for each batch.
Chuck
On 2014-05-05, 4:29 PM, "Tobias Janz" wrote:
Hi all,
i need to import a csv-file with about 58.000 Lines und 60 Columns.
I have 8 Entitys where the values will be stored. The conversion of
a line to entitys takes almost no time. But the full prozess, from
conversion until saveChanges is done, takes several hours (5-6h).
After profiling it points me to the generating of primary keys.
This part uses half of the time.
I need to do the import several times.
Do you see a way to improve the speed?
I'm using WebObject 5.4.3 (with Wonder). My Development is on Mac.
Database is a Frontbase in the cloud on centOS.
Thanks for any advice.
Tobias
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
40onestep.co.uk
This email sent to email@hidden
This email, including any attachments, is confidential and intended solely
for the person or organisation to whom it is addressed. If you are not the
intended recipient you must not disseminate, distribute or copy any part
of this email nor take any action in reliance on it.
If you have received this in error please notify the sender immediately by
email or phone +44 (0)1702 426400 and delete this email and any attachments
from your system.
Email transmission cannot be guaranteed to be secure or error-free as
information could be intercepted, corrupted, lost, destroyed, arrive
late or incomplete, or contain viruses. The sender therefore does not
accept liability for any errors or omissions in the contents of this
message which arise as a result of email transmission. If verification
is required please request a hard-copy version.
OneStep Solutions LLP is registered in England and Wales under registration
number OC337173 and has its registered office at 44 The Pantiles, Tunbridge
Wells, Kent, TN2 5TN.
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden