Apple

On Jun 30, 2011, at 3:38 PM, Anatol Pomozov wrote:

Hi,

I work on a build tool called tup, and most of the time it works fine. But unfortunately when I run tests it deadlocks on one of the open() syscalls. The probability that a test deadlocks is ~1%, but it always happens in different tests. I think deadlock happens here https://github.com/gittup/tup/blob/master/src/compat/dir_mutex.c#L23

Looking at the stacktrace that I got from stackshot I see that open() syscall calls vfs_mountroot (??) that tried to acquire mutex on a some node. It looks like the mutex is already hold by someone else. An interesting thing is that I see the deadlock in tests only, and I never saw it in the production. Maybe it happens because tests create a lot of short-living fuse filesystems (and call a bunch of mount/umount)?

I do not know whether this is an issue in the kernel. It can be also related to fuse4x kernel extension that my application uses http://github.com/fuse4x/kext

Are there any pointers that help me to understand what is going on here?

This is a stacktrace for 32 bits kernel (10.6.7) and 64 bits app:

PID: 40062

Process: tup
Kernel stack:
machine_switch_context (in mach_kernel) + 753 (0x2a8a6e)

thread_dispatch (in mach_kernel) + 1966 (0x226fbe)
thread_block_reason (in mach_kernel) + 331 (0x22725d)
thread_block (in mach_kernel) + 33 (0x2272eb)
lck_mtx_lock_wait_x86 (in mach_kernel) + 330 (0x29f85a)

lck_mtx_lock (in mach_kernel) + 504 (0x2995d8)
lock_fsnode (in mach_kernel) + 97 (0x2fa9bd)
VNOP_ACCESS (in mach_kernel) + 83 (0x2fe21b)
vfs_mountroot (in mach_kernel) + 1617 (0x2e314b)

kauth_authorize_action (in mach_kernel) + 80 (0x46ba55)
vnode_authorize (in mach_kernel) + 91 (0x2dd78f)
vn_open_auth (in mach_kernel) + 1066 (0x2f400f)

link (in mach_kernel) + 1592 (0x2ed0d3)
open_nocancel (in mach_kernel) + 243 (0x2ed581)
unix_syscall64 (in mach_kernel) + 617 (0x4f6075)

lo64_unix_scall (in mach_kernel) + 77 (0x2a144d)
User stack:
open (in libSystem.B.dylib) + 10 (0x7fff8684495e)
Continuation: kevent (in mach_kernel) + 97 (0x4783e9)

User stack:
kevent (in libSystem.B.dylib) + 10 (0x7fff8685312a)
_dispatch_queue_invoke (in libSystem.B.dylib) + 185 (0x7fff86854cd4)
_dispatch_worker_thread2 (in libSystem.B.dylib) + 252 (0x7fff868547fe)

_pthread_wqthread (in libSystem.B.dylib) + 353 (0x7fff86854128)
start_wqthread (in libSystem.B.dylib) + 13 (0x7fff86853fc5)
Continuation: semaphore_wait_continue (in mach_kernel) + 0 (0x22a3e1)

User stack:
semaphore_wait_signal_trap (in libSystem.B.dylib) + 10 (0x7fff8683a2e2)
dir_mutex_lock (in tup) + 17 (0x10005f570)
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Darwin-kernel mailing list (email@hidden)
This email sent to email@hidden

References:
	>Deadlock in open() syscall (From: Anatol Pomozov <email@hidden>)
	>Re: Deadlock in open() syscall (From: Shantonu Sen <email@hidden>)