• Open Menu Close Menu
  • Apple
  • Shopping Bag
  • Apple
  • Mac
  • iPad
  • iPhone
  • Watch
  • TV
  • Music
  • Support
  • Search apple.com
  • Shopping Bag

Lists

Open Menu Close Menu
  • Terms and Conditions
  • Lists hosted on this site
  • Email the Postmaster
  • Tips for posting to public mailing lists
Re: PrimaryKey generation takes a lot of time
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: PrimaryKey generation takes a lot of time

  • Subject: Re: PrimaryKey generation takes a lot of time
  • From: Tobias Janz <email@hidden>
  • Date: Wed, 07 May 2014 09:21:52 +0200
Hi all,

First i've tryed to set the size of primaryKeys fetched per statement in _FrontBasePlugin to 500. I've gained abound 5%.
Next i've tryed to use ERXJDBCAdopter with batchsize 1000, but got no benefit. Then I tryed connectionpooling, but ended with a lot of exceptions, because the initialisation of Connections got broken. I switched to ObjectStoreCoordinator and the performance got worse.

Multithreading did also slow down the application (With 2 Threads i got a better performance by about 3-5%, but with more threads the performance slows down by 5% with 10 Threads and 15% with 20 Threads).

My solution:
1. Return the hole multithreading, ObjectStoreCoordinator.
2. Configure Testinstance on Productionsystem only to import the data. Database near Applikation got me to speed up. the 5 hours import is done in 5min.

Its not the best for me, but with the latency and slow connection issues it's the best i can get.

Thanks for all your hints.

regards
Tobias


2014-05-06 10:40 GMT+02:00 Patrick Middleton <email@hidden>:
We encountered this recently against MySQL 5.6.12.
(Not using Project Wonder)
The default primary key generation was invoking a stored procedure once per row.  Once per second, some sort of garbage collection would occur inside the RDBMS and the stored procedure would take about 0.2s to complete instead of about 0.005s.

The fix is to create and cache all your new primary keys at once in advance using something like
er.extensions.jdbc.ERXJDBCAdaptor.Channel.primaryKeysForNewRowsWithEntity(int cnt, com.webobjects.eoaccess.EOEntity entity)

then use something like
databaseContextNewPrimaryKey(EODatabaseContext dbCtxt, Object object, EOEntity entity)
http://wocommunity.org/documents/javadoc/WebObjects/5.4.2/com/webobjects/eoaccess/EODatabaseContext.Delegate.html?is-external=true

to fetch a candidate new key from your cache rather than make a database round-trip to make one.


On 6 May 2014, at 08:20, Tobias Janz wrote:

I forgott to replay to the list.

the database is close to the server. I did not find the way to copy the Database from Mac OS Frontbase to CentOS Frontbase or the other way. I tryed to recover me lokal Database with a Backup created with CentOS-Frontbase, but it fails.

So I think the best way for me will be the WRITE ALL export.

I will take a look at this.

Thanks

Tobias


2014-05-06 7:58 GMT+02:00 Chuck Hill <email@hidden>:
 Hi Tobias,

I missed the "Frontbase in the cloud on centOS” before.  Is the app server not “close” to the database server?  With latency like this, you are going to face production issues, too.  That would be my first concern.

Some ideas to consider:
Import it locally, do a  WRITE ALL export and import on the server
Change to the binary UUID primary keys that are generated in the application
ERXLongPrimaryKeyFactory
Chuck


On 2014-05-05, 10:28 PM, "Tobias Janz" wrote:

Hi Chuck,

thanks for your advice, but this approch i have allready tryed. I create a new editing context for each 100. Also i've tryed multiple threads with relevant success.

When i do the import on my lokal database its much much faster. I think it's because of the latency. Is there a way to savely create primary key with multiple DB-connections in multiple Threads? And how do i get multiple database connections?

regards Tobias


2014-05-06 1:32 GMT+02:00 Chuck Hill <email@hidden>:
Are you creating the objects and saving them all at once?  If so, that is your problem.  Save them in batches of 500 or 1000 and it will go much quicker.  You might want to try making a new editing context for each batch.


Chuck


On 2014-05-05, 4:29 PM, "Tobias Janz" wrote:

Hi all,

i need to import a csv-file with about 58.000 Lines und 60 Columns. I have 8 Entitys where the values will be stored. The conversion of a line to entitys takes almost no time. But the full prozess, from conversion until saveChanges is done, takes several hours (5-6h). After profiling it points me to the generating of primary keys. This part uses half of the time.

I need to do the import several times.
Do you see a way to improve the speed?
I'm using WebObject 5.4.3 (with Wonder). My Development is on Mac. Database is a Frontbase in the cloud on centOS.

Thanks for any advice.

Tobias


 _______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list      (email@hidden.com)
Help/Unsubscribe/Update your Subscription:

This email sent to email@hidden


This email, including any attachments, is confidential and intended solely
for the person or organisation to whom it is addressed. If you are not the
intended recipient you must not disseminate, distribute or copy any partof this email nor take any action in reliance on it.

If you have received this in error please notify the sender immediately by
email or phone +44 (0)1702 426400 and delete this email and any attachments
from your system.

Email transmission cannot be guaranteed to be secure or error-free as
information could be intercepted, corrupted, lost, destroyed, arrive
late or incomplete, or contain viruses. The sender therefore does not
accept liability for any errors or omissions in the contents of this
message which arise as a result of email transmission. If verification
is required please request a hard-copy version.

OneStep Solutions LLP is registered in England and Wales under registration
number OC337173 and has its registered office at 44 The Pantiles, Tunbridge
Wells, Kent, TN2 5TN.

_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list      (email@hidden.com)
Help/Unsubscribe/Update your Subscription:

This email sent to email@hidden

 _______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list      (email@hidden)
Help/Unsubscribe/Update your Subscription:

This email sent to email@hidden
References: 
 >PrimaryKey generation takes a lot of time (From: Tobias Janz <email@hidden>)
 >Re: PrimaryKey generation takes a lot of time (From: Chuck Hill <email@hidden>)
 >Re: PrimaryKey generation takes a lot of time (From: Tobias Janz <email@hidden>)
 >Re: PrimaryKey generation takes a lot of time (From: Patrick Middleton <email@hidden>)

  • Prev by Date: Re: Wonder documentation on jenkins.wocommunity.org broken?
  • Next by Date: Re: ERXKeyFilter.setAnonymousUpdateEnabled(Z)V on remote server
  • Previous by thread: Re: PrimaryKey generation takes a lot of time
  • Next by thread: Re: AjaxGrid single row select UPDATE
  • Index(es):
    • Date
    • Thread