More instability in Distributed Objects used between multithreaded processes?
More instability in Distributed Objects used between multithreaded processes?
- Subject: More instability in Distributed Objects used between multithreaded processes?
- From: Dave Cox <email@hidden>
- Date: Fri, 21 Nov 2008 22:16:07 -0800
This is the fifth question in a series.
I am writing an application that uses Distributed Objects for communications among
several processes. I developed and tested on OSX 10.4.11 for several months, and
all seemed well. A few weeks back I finally upgraded to 10.5.5, and now I'm having
a variety of problems. The app is complex and getting more so, so I'm writing
simple programs to try to demonstrate the problems I encounter and rule out errors
in my own code obscured by all the complexity.
In this case, I have two processes 'client' and 'server' that want to talk to each
other. They are essentially peers, but 'server' runs most of the time while
'client' runs intermittently. These do not directly establish NSConnections each to
the other. Instead, there is a middle-man process ('broker') which vends a broker
object as the root of an NSConnection on a mach port with a well-known name. The
server and the client both connect first to the broker object and register their
peer objects with the broker. The server waits for the broker to notify it that the
client has registered, by returning a proxy to an object the broker has fetched from
the client. The client asks the broker immediately to return a proxy to an object
it has fetched from the server. When each has its proxy from the broker, it calls
through the proxy to a method of the other's peer object.
Thus both client and server are simultaneously communicating with the broker, and
Distributed Objects is doing its magic to establish (from either/both ends) a (pair
of) NSConnection between client and server. Also, when a connection is established
in either client or server process, it calls -runInNewThread on the connection so
there is a thread dedicated to handling incoming remote calls on the connection.
The server calls to the client object at most once; the client calls to the server
object a configurable number of times (I generally used 2). Then the client
terminates and the server goes back to waiting for the client to connect. I use a
shell script to run the client repeatedly.
There isn't any timing coordination between server and client, other than the server
waiting for the client to get connected to the broker, so the exact sequence of
events can vary, but regardless, it generally works. It might work 99 times out of
a hundred. But then, deep inside Distributed Objects classes and supporting code,
exceptions are thrown, errors are logged, or on occasion, a call into a proxy never
returns. Either client or server may experience the exception. And the logged
error message may or may not be followed by the exception.
Below is the code, and following that some representative output and crash dumps. I
apologize for the length, but I already deleted most of the error handling code to
try to trim it down.
--------------------------------------------------------------------------------
// shared.h
#ifndef __SHARED_H__
#define __SHARED_H__
#import <Foundation/NSObject.h>
#import <Foundation/NSAutoreleasePool.h>
#import <Foundation/NSPort.h>
#import <Foundation/NSConnection.h>
#import <Foundation/NSPortNameServer.h>
#import <Foundation/NSDistantObject.h>
#import <Foundation/NSNotification.h>
#import <Foundation/NSRunLoop.h>
#import <Foundation/NSException.h>
#import <Foundation/NSString.h>
#import <Foundation/NSThread.h>
#import <Foundation/NSLock.h>
#import <unistd.h>
#import <ext/hash_map>
#import "log.h"
#define BROKER_NAME "PeerBroker"
// interface to server 'feature'
@protocol IFeature <NSObject>
- (int) exercise: (int) delay; // in microseconds
@end
// interface to server
@protocol IPeer <NSObject>
- (int) getFeature: (out byref id<IFeature>*) feature
getId: (out unsigned*) ident;
@end
typedef id<IPeer> TPeer;
// interface to broker
@protocol IBroker <NSObject>
- (int) registerServer: (in byref id<IPeer>) server;
- (int) waitForClientFeature: (out byref id<IFeature>*) feature
getId: (out unsigned*) ident;
- (int) registerClient: (in byref id<IPeer>) client;
- (int) getServerFeature: (out byref id<IFeature>*) feature
getId: (out unsigned*) ident;
@end
// feature class (client and server have distinct implementations)
@interface CFeature : NSObject <IFeature> {
@public
unsigned ident;
}
- (id) init;
- (void) dealloc;
@end
// connection listener (for init & died notifications)
// (client and server have distinct implementations)
@interface CConnectionListener : NSObject {}
- (void) onConnectionInit: (NSNotification*) notif;
- (void) onConnectionDied: (NSNotification*) notif;
@end
#endif // ndef'd __SHARED_H__
--------------------------------------------------------------------------------
// server.mm
#import "shared.h"
unsigned currentIdentity = 10000;
@implementation CFeature
- (id) init {
self = [super init];
if (self != nil) {
ident = currentIdentity++;
log("server feature init %u\n", ident);
}
return self;
}
- (void) dealloc {
log("server feature dealloc %u\n", ident);
[super dealloc];
}
- (int) exercise: (int) delay { // microseconds
log(">>> server exercise %u\n", ident);
usleep(delay);
return 1;
}
@end // @implementation CFeature
// server class
@interface CServer : NSObject <IPeer> {}
@end
@implementation CServer
- (int) getFeature: (out byref id<IFeature>*) feature
getId: (out unsigned*) ident {
CFeature* f = [[[CFeature alloc] init] autorelease];
*ident = f->ident;
*feature = f;
return 1;
}
@end // @implementation CServer
@implementation CConnectionListener
- (void) onConnectionInit: (NSNotification*) notif {
NSConnection* connection = (NSConnection*)[notif object];
log("connection init: %p\n", connection);
[connection runInNewThread];
}
- (void) onConnectionDied: (NSNotification*) notif {
NSConnection* connection = (NSConnection*)[notif object];
log("connection died: %p\n\n", connection);
}
@end
//----------------------------------------------------------------------------//
int main (int argc, char * const argv[]) {
if (argc < 2) {
log("not enough args\n");
return 1;
}
log("server...\n");
bool extraRelease = (argc > 1 && argv[1][0] == 'x');
NSAutoreleasePool* pool = [[NSAutoreleasePool alloc] init];
// register a connection init/died listener
CConnectionListener* listener
= [[[CConnectionListener alloc] init] autorelease];
[[NSNotificationCenter defaultCenter]
addObserver: listener
selector: @selector(onConnectionInit:)
name: NSConnectionDidInitializeNotification
object: nil ];
[[NSNotificationCenter defaultCenter]
addObserver: listener
selector: @selector(onConnectionDied:)
name: NSConnectionDidDieNotification
object: nil ];
CServer* me = [[[CServer alloc] init] autorelease];
// connect to broker object
NSMachBootstrapServer* namesrv = [NSMachBootstrapServer sharedInstance];
NSMachPort* brokerPort = (NSMachPort*)[namesrv portForName: @BROKER_NAME];
NSMachPort* receivePort = [[[NSMachPort alloc] init] autorelease];
NSConnection* connection
= [ NSConnection connectionWithReceivePort: receivePort
sendPort: brokerPort ];
[connection enableMultipleThreads];
id<IBroker> broker = (id<IBroker>)[connection rootProxy];
// register server object with broker
[broker registerServer: me];
// loop: wait for client to call
NSAutoreleasePool* innerPool = nil;
while (true) {
innerPool = [[NSAutoreleasePool alloc] init];
id<IFeature> feature = nil;
unsigned fid = 0;
@try {
// ask broker to wait for client to connect to it,
// get a feature from the client, and return it.
log("waitForClientFeature >>>\n");
int result = [broker waitForClientFeature: &feature getId: &fid];
log("waitForClientFeature <<<\n");
log("exercise %u >>>\n", fid);
result = [feature exercise: 1000];
if (extraRelease) {
[feature release];
}
} @catch (NSException* x) {
log("exception calling client: %s\n", [[x reason] UTF8String]);
} @finally {
[innerPool release];
innerPool = nil;
}
} // while
[pool release];
return 0;
}
--------------------------------------------------------------------------------
// client.mm
#import "shared.h"
unsigned currentIdentity = 0;
@implementation CFeature
- (id) init {
self = [super init];
if (self != nil) {
ident = currentIdentity++;
log("client feature init %u\n", ident);
}
return self;
}
- (void) dealloc {
log("client feature dealloc %u\n", ident);
[super dealloc];
}
- (int) exercise: (int) delay { // microseconds
log(">>> client exercise %u\n", ident);
usleep(delay);
return 1;
}
@end // @implementation CFeature
// server class
@interface CClient : NSObject <IPeer> {}
@end
@implementation CClient
- (int) getFeature: (out byref id<IFeature>*) feature
getId: (out unsigned*) ident {
CFeature* f = [[[CFeature alloc] init] autorelease];
*ident = f->ident;
*feature = f;
return 1;
}
@end // @implementation CServer
@implementation CConnectionListener
- (void) onConnectionInit: (NSNotification*) notif {
NSConnection* connection = (NSConnection*)[notif object];
log("connection init: %p\n", connection);
[connection runInNewThread];
}
- (void) onConnectionDied: (NSNotification*) notif {
NSConnection* connection = (NSConnection*)[notif object];
log("connection died: %p\n", connection);
}
@end
//----------------------------------------------------------------------------//
int main (int argc, char * const argv[]) {
if (argc < 4) {
printf("not enough args.\n");
return 1;
}
log("client...\n");
bool extraRelease = (argc > 1 && argv[1][0] == 'x');
int repetitions = atoi(argv[2]);
currentIdentity = (unsigned)atoi(argv[3]);
NSAutoreleasePool* pool = [[NSAutoreleasePool alloc] init];
// register a connection init/died listener
CConnectionListener* listener
= [[[CConnectionListener alloc] init] autorelease];
[[NSNotificationCenter defaultCenter]
addObserver: listener
selector: @selector(onConnectionInit:)
name: NSConnectionDidInitializeNotification
object: nil ];
[[NSNotificationCenter defaultCenter]
addObserver: listener
selector: @selector(onConnectionDied:)
name: NSConnectionDidDieNotification
object: nil ];
CClient* me = [[[CClient alloc] init] autorelease];
// connect to broker object
NSMachBootstrapServer* namesrv = [NSMachBootstrapServer sharedInstance];
NSMachPort* brokerPort = (NSMachPort*)[namesrv portForName: @BROKER_NAME];
NSMachPort* receivePort = [[[NSMachPort alloc] init] autorelease];
NSConnection* connection
= [ NSConnection connectionWithReceivePort: receivePort
sendPort: brokerPort ];
[connection enableMultipleThreads];
id<IBroker> broker = (id<IBroker>)[connection rootProxy];
// register peer object with broker
[broker registerClient: me];
// loop: call server...
NSAutoreleasePool* innerPool = nil;
while (repetitions-- > 0) {
// in case it didn't get released in previous loop
[innerPool release];
// new autorelease pool
innerPool = [[NSAutoreleasePool alloc] init];
id<IFeature> feature = nil;
unsigned fid = 0;
// let debugger/CrashReporter catch and analyze exceptions
//@try {
// ask broker to get feature from server and return it.
int result = [broker getServerFeature: &feature getId: &fid];
log("exercise %u >>>\n", fid);
result = [feature exercise: 1000];
if (extraRelease) {
[feature release];
}
//} @catch (NSException* x) {
// log("exception calling server: %s\n", [[x reason] UTF8String]);
//} @finally {
[innerPool release];
innerPool = nil;
//}
} // while
log("quitting\n\n");
usleep(100000);
[pool release];
return 0;
}
--------------------------------------------------------------------------------
// broker.mm
#import "shared.h"
#import "CoreServices/CoreServices.h"
// broker class
@interface CBroker : NSObject <IBroker> {
TPeer theServer;
TPeer theClient;
MPEventID clientRegisteredEvent;
NSRecursiveLock* lock;
}
- (id) init;
- (void) dealloc;
@end
@implementation CBroker
- (id) init {
self = [super init];
if (self != nil) {
theServer = nil;
theClient = nil;
lock = [[NSRecursiveLock alloc] init];
(void)MPCreateEvent(&clientRegisteredEvent);
}
return self;
}
- (void) dealloc {
MPDeleteEvent(clientRegisteredEvent);
[lock release];
[super dealloc];
}
- (int) registerServer: (id<IPeer>) server {
[lock lock];
@try {
[server retain];
id<IPeer> old = theServer;
[old release];
theServer = server;
log("server %p registered\n", server);
} @catch (NSException* x) {
log("registerServer: exception: %s\n", [[x reason] UTF8String ]);
} @finally {
[lock unlock];
}
return 1;
}
- (int) getServerFeature: (out byref id<IFeature>*) feature
getId: (out unsigned*) ident {
[lock lock];
*feature = nil;
id<IPeer> server = nil;
// see if the requested server is registered and if so, retain it so
// it can't be destroyed on another thread once we release the lock.
@try {
server = theServer;
if (server != nil) {
[[server retain] autorelease];
}
} @catch (NSException* x) {
log("getServerFeature: exception: %s\n", [[x reason] UTF8String]);
return 1;
} @finally {
[lock unlock];
}
if (server != nil) {
@try {
// now call the server to get its feature, return it.
id<IFeature> f = nil;
[server getFeature: &f getId: ident];
*feature = f;
if (true) { // extraRelease
[f release];
}
} @catch (NSException* x) {
log("getServerFeature: exception: %s\n", [[x reason] UTF8String]);
}
}
return 1;
}
- (int) registerClient: (id<IPeer>) client {
[lock lock];
@try {
[client retain];
id<IPeer> old = theClient;
[old release];
theClient = client;
log("client %p registered\n", client);
// notify waiting server that client is connected
MPSetEvent(clientRegisteredEvent, 1);
} @catch (NSException* x) {
log("registerClient: exception: %s\n", [[x reason] UTF8String ]);
} @finally {
[lock unlock];
}
return 1;
}
- (int) waitForClientFeature: (out byref id<IFeature>*) feature
getId: (out unsigned*) ident {
log("waitForClientFeature: waiting...\n\n");
// wait until signaled by registerClient:
MPWaitForEvent(clientRegisteredEvent, NULL, kDurationForever);
log("waitForClientFeature: done waiting.\n");
[lock lock];
*feature = nil;
id<IPeer> client = nil;
// see if the requested client is registered and if so, retain it so
// it can't be destroyed on another thread once we release the lock.
@try {
client = theClient;
if (client != nil) {
// ask the proxy if it's valid?
[[client retain] autorelease];
}
} @catch (NSException* x) {
log("waitForClientFeature: exception: %s\n", [[x reason] UTF8String]);
return 1;
} @finally {
[lock unlock];
}
if (client != nil) {
@try {
// now call the client to get its feature, return it.
id<IFeature> f = nil;
[client getFeature: &f getId: ident];
*feature = f;
if (true) { // extraRelease
[f release];
}
} @catch (NSException* x) {
log( "waitForClientFeature: exception: %s\n",
[[x reason] UTF8String] );
}
}
return 1;
}
@end // implementation CBroker
//----------------------------------------------------------------------------//
int main (int argc, char * const argv[]) {
log("broker...\n");
NSAutoreleasePool* pool = [[NSAutoreleasePool alloc] init];
NSMachPort* port = nil;
NSConnection* connection = nil;
CBroker* broker = nil;
// vend broker object...
port = [[[NSMachPort alloc] init] autorelease];
connection = [NSConnection connectionWithReceivePort:port sendPort:nil];
[connection enableMultipleThreads];
// create broker object to vend
broker = [[[CBroker alloc] init] autorelease];
// vend object
[connection setRootObject: broker];
// publish name
NSMachBootstrapServer* namesrv = [NSMachBootstrapServer sharedInstance];
if ([namesrv registerPort: port name: @BROKER_NAME] == NO) {
log("broker: error: port not registered\n");
}
// detach new threads to run runloops for the connection
for (unsigned n = 0; n < 4; n++) {
[connection runInNewThread];
}
// run the server until terminated
while (true) {
sleep(10);
}
//[port invalidate];
[pool release];
return 0;
}
--------------------------------------------------------------------------------
*** Normal Output ***
--------------------------------------------------------------------------------
server:
20081121 14:29:32 [65843] <0xa065dfa0> server feature dealloc 10180
20081121 14:29:32 [65843] <0xa065dfa0> server feature init 10181
20081121 14:29:32 [65843] <0xa065dfa0> connection init: 0x135980
20081121 14:29:32 [65843] <0xa065dfa0> waitForClientFeature <<<
20081121 14:29:32 [65843] <0xa065dfa0> exercise 92 >>>
20081121 14:29:32 [65843] <0xa065dfa0> >>> server exercise 10181
20081121 14:29:32 [65843] <0xa065dfa0> server feature dealloc 10181
20081121 14:29:32 [65843] <0xb0081000> server feature init 10182
20081121 14:29:32 [65843] <0xa065dfa0> waitForClientFeature >>>
20081121 14:29:32 [65843] <0xa065dfa0> >>> server exercise 10182
20081121 14:29:32 [65843] <0xa065dfa0> connection died: 0x135980
20081121 14:29:32 [65843] <0xa065dfa0> server feature dealloc 10182
20081121 14:29:32 [65843] <0xa065dfa0> server feature init 10183
20081121 14:29:32 [65843] <0xa065dfa0> connection init: 0x137630
20081121 14:29:32 [65843] <0xa065dfa0> waitForClientFeature <<<
20081121 14:29:32 [65843] <0xa065dfa0> exercise 93 >>>
20081121 14:29:32 [65843] <0xb0081000> >>> server exercise 10183
20081121 14:29:32 [65843] <0xa065dfa0> waitForClientFeature >>>
20081121 14:29:32 [65843] <0xa065dfa0> server feature init 10184
20081121 14:29:32 [65843] <0xa065dfa0> server feature dealloc 10183
20081121 14:29:32 [65843] <0xa065dfa0> >>> server exercise 10184
20081121 14:29:32 [65843] <0xa065dfa0> connection died: 0x137630
client:
20081121 14:29:32 [65939] <0xa065dfa0> client...
20081121 14:29:32 [65939] <0xa065dfa0> connection init: 0x1071b0
20081121 14:29:32 [65939] <0xa065dfa0> connection init: 0x10b2d0
20081121 14:29:32 [65939] <0xa065dfa0> exercise 10181 >>>
20081121 14:29:32 [65939] <0xa065dfa0> client feature init 92
20081121 14:29:32 [65939] <0xa065dfa0> >>> client exercise 92
20081121 14:29:32 [65939] <0xa065dfa0> exercise 10182 >>>
20081121 14:29:32 [65939] <0xa065dfa0> client feature dealloc 92
20081121 14:29:32 [65939] <0xa065dfa0> quitting
20081121 14:29:32 [65941] <0xa065dfa0> client...
20081121 14:29:32 [65941] <0xa065dfa0> connection init: 0x1071b0
20081121 14:29:32 [65941] <0xa065dfa0> connection init: 0x10b2d0
20081121 14:29:32 [65941] <0xa065dfa0> exercise 10183 >>>
20081121 14:29:32 [65941] <0xb0081000> client feature init 93
20081121 14:29:32 [65941] <0xa065dfa0> >>> client exercise 93
20081121 14:29:32 [65941] <0xa065dfa0> exercise 10184 >>>
20081121 14:29:32 [65941] <0xa065dfa0> client feature dealloc 93
20081121 14:29:32 [65941] <0xa065dfa0> quitting
--------------------------------------------------------------------------------
*** Error (client crash) ***
--------------------------------------------------------------------------------
server:
20081121 14:29:32 [65843] <0xa065dfa0> server feature dealloc 10186
20081121 14:29:32 [65843] <0xa065dfa0> server feature init 10187
20081121 14:29:32 [65843] <0xa065dfa0> connection init: 0x138f80
20081121 14:29:32 [65843] <0xa065dfa0> waitForClientFeature <<<
20081121 14:29:32 [65843] <0xa065dfa0> exercise 95 >>>
20081121 14:29:33 [65843] <0xa065dfa0> connection died: 0x138f80
20081121 14:29:33 [65843] <0xa065dfa0> server feature dealloc 10187
20081121 14:29:33 [65843] <0xa065dfa0> exception calling client: connection went invalid while waiting for a reply
20081121 14:29:33 [65843] <0xa065dfa0> waitForClientFeature >>>
client:
20081121 14:29:32 [65943] <0xa065dfa0> client...
20081121 14:29:32 [65943] <0xa065dfa0> connection init: 0x1071a0
20081121 14:29:32 [65943] <0xa065dfa0> client feature init 95
20081121 14:29:32 [65943] <0xb0081000> connection init: 0x10e6e0
20081121 14:29:32 [65943] <0xa065dfa0> exercise 10187 >>>
20081121 14:29:32 [65943] <0xa065dfa0> client feature dealloc 95
2008-11-21 14:29:32.944 client[65943:807] *** NSDistantObject initWithCoder: 0x2 not given away for conn 0x10e6e0
2008-11-21 14:29:32.945 client[65943:807] *** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: '*** -[NSConcretePortCoder decodeInvocation]: no local target'
2008-11-21 14:29:32.946 client[65943:807] Stack: (
2456817995,
2484096571,
2456817451,
2456817514,
2502929276,
2502927901,
2502912780,
2502925358,
2502924621,
2502923390,
2502921187,
2456172085,
2456320264,
2456321272,
2502908429,
2502903730,
2456839869,
2456840722,
14456,
10911,
10701
)
Trace/BPT trap
CrashReporter:
Process: client [65933]
Path: ./client
Identifier: client
Version: ??? (???)
Code Type: X86 (Native)
Parent Process: tcsh [65844]
Date/Time: 2008-11-21 14:29:30.828 -0800
OS Version: Mac OS X 10.5.5 (9F33)
Report Version: 6
Exception Type: EXC_BREAKPOINT (SIGTRAP)
Exception Codes: 0x0000000000000002, 0x0000000000000000
Crashed Thread: 0
Application Specific Information:
*** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: '*** -[NSConcretePortCoder decodeInvocation]: no local target'
Thread 0 Crashed:
0 com.apple.CoreFoundation 0x92700ff4 ___TERMINATING_DUE_TO_UNCAUGHT_EXCEPTION___ + 4
1 libobjc.A.dylib 0x94104e3b objc_exception_throw + 40
2 com.apple.Foundation 0x952f57c6 -[NSConnection sendInvocation:internal:] + 2422
3 com.apple.Foundation 0x952f47b2 -[NSDistantObject methodSignatureForSelector:] + 1490
4 com.apple.CoreFoundation 0x927066bd ___forwarding___ + 237
5 com.apple.CoreFoundation 0x92706a12 _CF_forwarding_prep_0 + 50
6 client 0x00003878 main + 1054 (client.mm:189)
7 client 0x00002a9f _start + 209
8 client 0x000029cd start + 41
--------------------------------------------------------------------------------
*** Error (server exception, client hang) ***
--------------------------------------------------------------------------------
server:
20081121 14:27:30 [65797] <0xa065dfa0> server feature dealloc 10055
20081121 14:27:30 [65797] <0xa065dfa0> server feature init 10056
20081121 14:27:30 [65797] <0xa065dfa0> connection init: 0x120a80
20081121 14:27:30 [65797] <0xa065dfa0> waitForClientFeature <<<
20081121 14:27:30 [65797] <0xa065dfa0> exercise 29 >>>
2008-11-21 14:27:30.401 server[65797:807] *** NSDistantObject initWithCoder: 0x3a not given away for conn 0x120a80
20081121 14:27:30 [65797] <0xa065dfa0> exception calling client: *** -[NSConcretePortCoder decodeInvocation]: no local target
20081121 14:27:30 [65797] <0xa065dfa0> waitForClientFeature >>>
client:
20081121 14:27:30 [65828] <0xa065dfa0> client...
20081121 14:27:30 [65828] <0xa065dfa0> connection init: 0x1071a0
20081121 14:27:30 [65828] <0xa065dfa0> client feature init 29
20081121 14:27:30 [65828] <0xa065dfa0> connection init: 0x10ee20
20081121 14:27:30 [65828] <0xa065dfa0> exercise 10056 >>>
_______________________________________________
Cocoa-dev mailing list (email@hidden)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden