Re: Failed to provide primary keys?
Re: Failed to provide primary keys?
- Subject: Re: Failed to provide primary keys?
- From: Chuck Hill <email@hidden>
- Date: Wed, 14 Jul 2010 10:28:13 -0700
On Jul 14, 2010, at 6:39 AM, Kieran Kelleher wrote:
> Thanks for replying Chuck.
>
> This code is importing EOs from a text file in a background process. This code is used a *lot* and maybe after importing millions of EOs across multiple files/users in a week, I might get this error. Retrying an import will usually result in the same error. A restart of the instance will then usually "fix" the problem.
Have you checked for running out of memory?
> In development, I have used the same code to import 5 million records without this error, so the inability to reproduce this in development and the fact that it may happen once a week or once a month in deployment makes it a "important but not urgent" thorn in my side.
>
> The Setup: A pool of eight OSCs are reserved for background thread EOF processing to ensure no "delays" for normal page EOF activity. EditingContexts are recycled every 250 records imported. One instance is available as a different app name and generally has one user on that instance for production processing work and that does the bulk of large imports (mailing list data for fulfillment of online customer orders). This error always (IIRC) happens in that admin instance, so the admin app instance is not under heavy concurrent user load at all (though the database server might be).
>
> Here is where I catch the error (the IllegalStateException) and throw/report it. This is where the OSCs and ECs info is coming from (ERX*.outstandingLockDescription()). Any suggestions for additional state I should try and capture here to troubleshoot this:
>
> try {
> ec.saveChanges();
> } catch (EOGeneralAdaptorException e) {
> throw new RuntimeException(WKEOUtils.generalAdaptorExceptionInfo(e));
> } catch (IllegalStateException e) {
> // This is to try and gather enough info to solve the PITA occasional "failed to provide primary keys" error
> String msg = ExceptionUtils.getRootCauseMessage(e);
> StringBuilder b = new StringBuilder();
> b.append("Root Cause: ").append(msg).append("; ");
> b.append(ERXEC.outstandingLockDescription());
> b.append(ERXObjectStoreCoordinator.outstandingLockDescription());
> throw new RuntimeException(b.toString(), e);
> }
>
> Meanwhile, I have disabled ERXJDBCAdaptor on the admin instance as a first "process of elimination" step, and I may have to wait for a month to see if we get this error to even decide if that was a *probable* contributor.
I am not sure that disabling this is the path to proceed on. I'd be taking a look in there, in EOF, and in your database plugin to identify any potential failure points and then try to get logging in there to log if the failure happens.
> The entire group of machines (about 8 including DB/apache/other and a few WO) is on its own subnet with a fairly pricey Cisco router and while we have not had noticeable network issues at the time this happens or any other time.
Have you checked in the system logs for momentary network problems?
> I will park it for today since I have other "urgent" tasks today and get back to it after vacation, however one thing I have not done is retried the save. So I don't want to change too much in one go and then not know which change solved the issue. We will see how it does not with ERXJDBCAdaptor out of the equation.
>
> What I am really hoping for out of this discussion is a list of "state to capture and log" - I could even capture a bunch of state before I try and save, and then log that and more state if I get the error since it almost always happens in this one place in the code.
The symptoms look more to me like a low level database, JDBC, or network error that causes an exception deep in EOF. EOF is not handling this properly and getting into a confused state.
Chuck
> On Jul 13, 2010, at 1:15 PM, Chuck Hill wrote:
>
>>
>> On Jul 13, 2010, at 9:45 AM, Kieran Kelleher wrote:
>>
>>> Trying to track this down is like a needle in a haystack and I can never reproduce it in development. Any ideas on a strategy to capture the root cause?
>>>
>>> Error processing file: /tmp/JGZ36CHBY17ZGH7FH518CNCBY_BdrWithSuppression.txt. Erroneous results for import are: isSuccess = false;
>>> messages = null;
>>> userInfo = null
>>> java.lang.RuntimeException: Root Cause: IllegalStateException: Adaptor er.extensions.jdbc.ERXJDBCAdaptor@2ff88688 failed to provide new primary keys for entity 'CTDataRecord'; Currently 0 active ECs : {})No open editing contexts (of 0)0 active ObjectStoreCoordinators : {})No open ObjectStoreCoordinator (of 0)
>>
>> It seems odd that there are not OSCs and no ECs. Is that perhaps an indication of the root problem? Are these getting garbage collected out from under your tasks? Or are those messages just sketchy?
>>
>> The next thing I would do is to take a look in ERXJDBCAdaptor, JDBCAdaptor, and the plugin for your database to see why PK generation might fail.
>>
>> How often does this happen? Does it match up with any network outages between the app and database servers?
>>
>>
>> Chuck
>>
>>
>>> at wk.webobjects.eof.ImportEnterpriseObjectData.processFile(ImportEnterpriseObjectData.java:156)
>>> at wk.webobjects.eof.ImportEnterpriseObjectData.run(ImportEnterpriseObjectData.java:364)
>>> at wk.webobjects.eof.ImportEnterpriseObjectData.call(ImportEnterpriseObjectData.java:371)
>>> at wk.webobjects.eof.ImportEnterpriseObjectData.call(ImportEnterpriseObjectData.java:1)
>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:637)
>>> Caused by: java.lang.IllegalStateException: Adaptor er.extensions.jdbc.ERXJDBCAdaptor@2ff88688 failed to provide new primary keys for entity 'CTDataRecord'
>>> at com.webobjects.eoaccess.EODatabaseContext.prepareForSaveWithCoordinator(EODatabaseContext.java:5885)
>>> at com.webobjects.eocontrol.EOObjectStoreCoordinator.saveChangesInEditingContext(EOObjectStoreCoordinator.java:409)
>>> at com.webobjects.eocontrol.EOEditingContext.saveChanges(EOEditingContext.java:3226)
>>> at er.extensions.eof.ERXEC._saveChanges(ERXEC.java:1085)
>>> at er.extensions.eof.ERXEC.saveChanges(ERXEC.java:1007)
>>> at wk.webobjects.eof.ImportEnterpriseObjectData.processFile(ImportEnterpriseObjectData.java:147)
>>> ... 8 more
>>>
>>> _______________________________________________
>>> Do not post admin requests to the list. They will be ignored.
>>> Webobjects-dev mailing list (email@hidden)
>>> Help/Unsubscribe/Update your Subscription:
>>>
>>> This email sent to email@hidden
>>
>> --
>> Chuck Hill Senior Consultant / VP Development
>>
>> Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems.
>> http://www.global-village.net/products/practical_webobjects
>>
>>
>>
>>
>>
>>
>>
>
--
Chuck Hill Senior Consultant / VP Development
Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems.
http://www.global-village.net/products/practical_webobjects
Attachment:
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden