February 2009 - Darwin-dev - lists.apple.com

Re: Debugging a machine check panic
by Terry Lambert 13 Feb '09

13 Feb '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Definition: <http://en.wikipedia.org/wiki/Machine_Check_Exception> Using two machine debugging, you can enter the kgmacros command: (gdb) showMCAstate ...to dump the machine check state for your CPU(s). Then you can build a copy of mcelog from: <http://freshmeat.net/projects/mcelog/> -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Feb 12, 2009, at 2:46 PM, Ryan McGann wrote: I have an NKE that intermittently, but reliably, panics in what looks like an interrupt handler: panic(cu 0 caler 0x001A8Fd4): Machine-check at 0x0014329a, thread: 0x3639790, code:0x12, registers: CR0: 0x8001003b, CR2: 0x37d0bfec, CR3: 0x0107d000, CR4: 0x0000006e0 EAX: 0x00522d08, EBX: 0x00000000, ECX: 0x00000000, EDX: 0x00001000 ESP: 0x37d0bff0, EBP: 0x37d0c068, ESI: 0x00000000, EDI: 0x00522d08 EFL: 0x00010206, EIP: 0x0019f967 I can reproduce the panic fairly easily but I cannot see in the code where it is panicing. The EIP given in the panic log is someplace in hw_simplelock OR zalloc_canblock, but never in my code (though I'm sure it's my kext causing the panic). I just don't know how to tell where it's going wrong. I don't even know what this panic indicates since I don't know exactly what machine-check is, other than some sort of interrupt. My guess is stack overflow or stack corruption, but I could be dead wrong. The ESP and CR2 registers may be inside kext, since showallkmods shows my kext is loaded at 0x2bc7a000, though that'd be a large jump (the kext is 800k). I've done all the usual debugging that I can think of but two machine debugging is pretty useless since the backtrace is not in my code, and printf can only tell me the last function in my code that is called, not any kernel functions I call, (plus it's not as reliable as one would hope). I've commented out a lot of code to the point that it doesn't panic anymore, but also doesn't do anything useful anymore. So I'm pretty much at a dead end. Anybody have any other ideas on where to begin, like what machine- check means? ...basically, it thinks you have a hardware problem. There are typically two causes for something like this: (1) You have a hardware problem (usually, bad RAM that can't clock up fast enough - within manufacturers spec - after it's been clocked down during a power management operation) (2) You have software that's stomping saved state information that, when restored, causes your hardware to barf (3) You are trying to change the contents of MSRs, not realizing that these are adjusted dynamically by the OS for architectural reasons, and must be left alone by third parties ...I'm going to guess #2 for this one, given the information you've provided so far. ...and shovel the register data through there, and hope it tells you something Andi Kleen was able to figure out from the architecture manuals, excluding the Intel NDA'd information, so you can figure out what you're causing to be stomped. Alternately you can file a bug report, and hope you can gather enough information that whoever it lands on can do this for you, or you could probably find the problem if you were to go through your NKE with a fine tooth comb doing a source inspection for logic errors. This email sent to site_archiver(a)lists.apple.com

1 0

How Mutexes Work
by Dave Keck 13 Feb '09

13 Feb '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:date :x-google-sender-auth:message-id:subject:from:to:content-type :content-transfer-encoding; bh=zTmWymTJg9bQ5kmasEpH1l/Am8DzZtCQS+6vFKpqFTk=; b=N8WuStk/stP/CDaDRuXBHaeYJ8qpS5O7A1ZVHZSBE5UcREMzuyQ0CcR+C8H0vEKVO4 Kqxitg1HDQJbBhCxNRSwbxQPKYj+iWiapD7BywxCPyL3fXvVit09iSRYeWljOl6whYBl 5aqJRQpM3MBa1qL7NQkt0KE7rzKLxWG7zgfB4= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:content-type:content-transfer-encoding; b=D77jv3tOKGpxtBk9vwj6VR6zJH9eoFe0ULxyuTUhDODKp4dCXubFbpM1tOhKuEIrUi QgGP7SgoValXhVMbE4RQGeeCxfCYOpfp3GYm6Skw7LrgrOoCFc84voywoyQG4Xyy8hU9 HSw0g5a9HNOAh7IOEFrIEgGliI0muIISGXlL8= Hey, Today, for some reason or another, I've been pondering how locks work. After locating the pthread source (in libc, available at opensource.apple.com, for those curious) I've found the source for the pthread_mutex_lock, in pthread_mutex.c It seems the meat of this function are the macros 'LOCK' and 'PLOCKSTAT_MUTEX_ACQUIRE'. The 'LOCK' macro is defined (indirectly) as _spin_lock. _spin_lock is implemented in various places based on the architecture you're compiling for; in my case I'm interested in i386. For that, _spin_lock is defined as: _OSSpinLockLock: _spin_lock: __spin_lock: movl $(_COMM_PAGE_SPINLOCK_LOCK), %eax jmpl *%eax .align 2, 0x90 .globl _OSSpinLockUnlock .globl _spin_unlock .globl __spin_unlock But I can't seem to find what _COMM_PAGE_SPINLOCK_LOCK is. I'm taking a shot in the dark that it's a function address in the kernel's address space. Is this correct? (I'm no assembly expert! :)) If so, where is the value for _COMM_PAGE_SPINLOCK_LOCK set? Secondly we have PLOCKSTAT_MUTEX_ACQUIRE. I can't seem to find where it's defined either, and it seems to be the real heavy-lifter for pthread_mutex_lock. Any ideas? Lastly and perhaps most important, is there a good low-level OS book that might explain the nitty gritty when it comes to locks and everything else deep under-the-hood? Thanks a lot! David _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Debugging a machine check panic
by Ryan McGann 12 Feb '09

12 Feb '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com I have an NKE that intermittently, but reliably, panics in what looks like an interrupt handler: panic(cu 0 caler 0x001A8Fd4): Machine-check at 0x0014329a, thread:0x3639790, code:0x12, registers: CR0: 0x8001003b, CR2: 0x37d0bfec, CR3: 0x0107d000, CR4: 0x0000006e0 EAX: 0x00522d08, EBX: 0x00000000, ECX: 0x00000000, EDX: 0x00001000 ESP: 0x37d0bff0, EBP: 0x37d0c068, ESI: 0x00000000, EDI: 0x00522d08 EFL: 0x00010206, EIP: 0x0019f967 I can reproduce the panic fairly easily but I cannot see in the code where it is panicing. The EIP given in the panic log is someplace in hw_simplelock OR zalloc_canblock, but never in my code (though I'm sure it's my kext causing the panic). I just don't know how to tell where it's going wrong. I don't even know what this panic indicates since I don't know exactly what machine-check is, other than some sort of interrupt. My guess is stack overflow or stack corruption, but I could be dead wrong. The ESP and CR2 registers may be inside kext, since showallkmods shows my kext is loaded at 0x2bc7a000, though that'd be a large jump (the kext is 800k). I've done all the usual debugging that I can think of but two machine debugging is pretty useless since the backtrace is not in my code, and printf can only tell me the last function in my code that is called, not any kernel functions I call, (plus it's not as reliable as one would hope). I've commented out a lot of code to the point that it doesn't panic anymore, but also doesn't do anything useful anymore. So I'm pretty much at a dead end. Anybody have any other ideas on where to begin, like what machine-check means? Thanks, Ryan ----------------------- Ryan McGann rmcgann(a)mac.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: query on stack trace
by Terry Lambert 12 Feb '09

12 Feb '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

query on stack trace
by Arnab Ganguly 12 Feb '09

12 Feb '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=2ciXRhEJkqwyjfUbnfjwlAmNNXGWfSyO9rH3a8Hg8LI=; b=c5P8NTB//crQt2H2nCyrsJEFhRxhEyhKuOybKiZo22tWNAdFeEEaSTLUNVGuAURZjD jVpZinSNMGAdqNOBEI3i0V2vySunfrbNmpZXGz8ls35OMyTB9BGx0uXtngyRJ6tAqbuI sI9N9epRxgVKKucDeYxxkdz4ydxLAlVi5581o= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=bBRhFE/42s/XsinNGQecZe7eefq2Wwz5r8xP2yGXgoe/iWf4qUHrSgtrGQVOxpicGX hJ+cMUXABjh/aHDHoIEIgk+wTLWSK0SBnb22WphtzMSV+4RAJKHCjiA2IEws/iHDP14R fYUr1YzR1LTqNO/mt8QSfgxmu5U5OnDKVVx0A= _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Darwin on an Asus eeePC
by Eli Bach 12 Feb '09

12 Feb '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:from:to :in-reply-to:content-type:content-transfer-encoding:subject :mime-version:date:references:x-mailer; bh=LvDWUTSc7P3hPTFgSJZArUhIlkUSJC3bQIo52KYLoxc=; b=FRlmoFTcmXzH8VwC1dCfXBw99b00EVojMjcDWYPOlehQMRbn9JwMX3GsCRS7f+UfY7 XTm1FaLER+SGYkYyOy+/KVWMuotaVj/1eHuRw1lX8+AB03U7jj5hJIdRktVQzZ9ohqoU dgE3qTN3uO5jvASZavLCzFdTeBqp6KTL6O1Hw= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:from:to:in-reply-to:content-type :content-transfer-encoding:subject:mime-version:date:references :x-mailer; b=vlU/30YQb04ehvuVTWDTGb72mAzSVJNDy+2/MUNishcZyIzGhHCU+bGn0YmZMO/k7G tsXqnBhJqwQdGx6ioTDtHMfSaRQ/5ltOE0JjDrFr1dh4MeuTG/96i4AdX7qCYRkSLv9B i1roZM92C18GVXHULJwOur0dFap3NzwwKf+ro= On Feb 11, 2009, at 4:52 PM, Jason Todd Slack-Moehrle wrote: Hi All, Does Darwin on an Asus eee PC does anyone know? Are there Window/Session Managers for Darwin? Thanks! -Jason No. Eli _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Second, since Darwin is the foundation of OS X, should I be able to run and/or develop OS X applications? Same API as on OS X? The GUI frameworks that provide the 'Mac' experience aren't part of Darwin. You'll need the full Mac OS X install to reasonably be able to create and test Mac OS X GUI apps. If you application is not GUI based, then you could use Darwin alone (probably). You may be able to get X-Windows going on Darwin, but that's a completely different environment than what Mac OS X uses. Given that some people have hacked Mac OS X to run on these eee pc's (I believe I saw it on a web a couple of months ago), Darwin should also run, but I have no idea what kind of effort will be required. This email sent to site_archiver(a)lists.apple.com

1 0

Darwin on an Asus eeePC
by Jason Todd Slack-Moehrle 12 Feb '09

12 Feb '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Hi All, Does Darwin on an Asus eee PC does anyone know? Are there Window/Session Managers for Darwin? Thanks! -Jason _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Second, since Darwin is the foundation of OS X, should I be able to run and/or develop OS X applications? Same API as on OS X? This email sent to site_archiver(a)lists.apple.com

1 0

Re: Documentation for values of Previous Sleep and Shutdown Cause
by Ethan Bold 11 Feb '09

11 Feb '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Strange "hang" in pthread_cond_destroy
by Terry Lambert 11 Feb '09

11 Feb '09

site_archiver(a)lists.apple.com Delivered-To: Darwin-dev(a)lists.apple.com -- Terry On Feb 11, 2009, at 7:00 AM, Stéphane Letz <letz(a)grame.fr> wrote: Hi, Sampling process 115 for 3 seconds with 1 millisecond of run time between samples Sampling completed, processing symbols... Analysis of sampling coreaudiod (pid 115) every 1 millisecond Call graph: 2473 Thread_2503 2473 0xbfffff38 2473 0x24cc 2473 0xae60 2473 AudioHardwareBeMaster 2473 HALSystem::CheckOutInstance() 2473 HALSystem::InitializeDevices() 2473 HALPlugInManagement::Initialize() 2473 HALPlugInManagement::CreateHALPlugIn(HALCFPlugIn const*) 2473 __ZL40HP_HardwarePlugIn_InitializeWithObjectIDPP28AudioHardwarePlugInInterfacem 2473 JackRouterPlugIn::InitializeWithObjectID(unsigned long) 2473 JackRouterPlugIn::CheckServer(unsigned long) 2473 jack_client_open 2473 jack_client_open_aux 2473 Jack::JackMessageBuffer::Destroy() 2473 Jack::JackMessageBuffer::~JackMessageBuffer() 2473 pthread_cond_destroy 2211 __spin_lock_relinquish 2211 __spin_lock_relinquish 262 __spin_lock 262 __spin_lock Total number in stack (recursive counted multiple, when >=5): Sort by top of stack, same collapsed (when >= 5): __spin_lock_relinquish 2211 __spin_lock 262 Sample analysis of process 115 written to file /dev/stdout pthread_cond_t fCond; pthread_cond_init(&fCond, NULL); What could be the cause of this hang? Thanks Stephane Letz _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/tlambert%40apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Sample only gives statistical profiling; that is, it only gives information about where the program counter happened to be at the time you were looking. As such, you're not going to get a literal call graph like you would with, for example, a dtruss to a gdb bt. The graph you show could just as easily be you repeatedly entering the same code, or making a call that blocks a long time. My guess would be a retry loop at the InitializeWithObjectID, probably at JackRouterPlugIn, or up one level, depending on the return code and how the code above it deals with a bad return with that code. An east way to check this is to see if it's actually blocked by looking at how much CPU it's using (dtruss will only give information on it making real System calls; dtrace in user space might give more information). We get a strange "hang" but report in our code where it seems that "pthread_cond_destroy" is hanging: OS Version: 10.5.6 (Build 9G55) Model: PowerBook6,4, BootROM 4.8.3f1, 1 processor, PowerPC G4 (1.1), 1.33 GHz, 1,25 GB Graphics: GeForce FX Go5200, GeForce FX Go5200, spdisplays_agp_device, 64 MB Memory Module: DIMM0/BUILT-IN, 256 MB, built-in, built-in Memory Module: DIMM1/J31, 1 GB, DDR SDRAM, PC2700U-25330 AirPort: spairport_wireless_card_type_airport_extreme (0x14E4, 0x4E), Broadcom BCM43xx 1.0 (4.170.25.8) Modem: Jump, V.92, Version 1.0 Bluetooth: Version 2.1.3f8, 2 service, 0 devices, 1 incoming serial ports Network Service: AirPort, AirPort, en1 Parallel ATA Device: MATSHITACD-RW CW-8123 Parallel ATA Device: ST980815A, 74,53 GB USB Device: Bluetooth USB Host Controller, (null) mA We checked again that the condition variable is correctly allocated with this kind of code: This email sent to tlambert(a)apple.com This email sent to site_archiver(a)lists.apple.com

1 0

Strange "hang" in pthread_cond_destroy
by Stéphane Letz 11 Feb '09

11 Feb '09

site_archiver(a)lists.apple.com Delivered-To: Darwin-dev(a)lists.apple.com Hi, Sampling process 115 for 3 seconds with 1 millisecond of run time between samples Sampling completed, processing symbols... Analysis of sampling coreaudiod (pid 115) every 1 millisecond Call graph: 2473 Thread_2503 2473 0xbfffff38 2473 0x24cc 2473 0xae60 2473 AudioHardwareBeMaster 2473 HALSystem::CheckOutInstance() 2473 HALSystem::InitializeDevices() 2473 HALPlugInManagement::Initialize() 2473 HALPlugInManagement::CreateHALPlugIn(HALCFPlugIn const*) 2473 __ZL40HP_HardwarePlugIn_InitializeWithObjectIDPP28AudioHardwarePlugInInterfacem 2473 JackRouterPlugIn::InitializeWithObjectID(unsigned long) 2473 JackRouterPlugIn::CheckServer(unsigned long) 2473 jack_client_open 2473 jack_client_open_aux 2473 Jack::JackMessageBuffer::Destroy() 2473 Jack::JackMessageBuffer::~JackMessageBuffer() 2473 pthread_cond_destroy 2211 __spin_lock_relinquish 2211 __spin_lock_relinquish 262 __spin_lock 262 __spin_lock Total number in stack (recursive counted multiple, when >=5): Sort by top of stack, same collapsed (when >= 5): __spin_lock_relinquish 2211 __spin_lock 262 Sample analysis of process 115 written to file /dev/stdout pthread_cond_t fCond; pthread_cond_init(&fCond, NULL); What could be the cause of this hang? Thanks Stephane Letz _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… We get a strange "hang" but report in our code where it seems that "pthread_cond_destroy" is hanging: OS Version: 10.5.6 (Build 9G55) Model: PowerBook6,4, BootROM 4.8.3f1, 1 processor, PowerPC G4 (1.1), 1.33 GHz, 1,25 GB Graphics: GeForce FX Go5200, GeForce FX Go5200, spdisplays_agp_device, 64 MB Memory Module: DIMM0/BUILT-IN, 256 MB, built-in, built-in Memory Module: DIMM1/J31, 1 GB, DDR SDRAM, PC2700U-25330 AirPort: spairport_wireless_card_type_airport_extreme (0x14E4, 0x4E), Broadcom BCM43xx 1.0 (4.170.25.8) Modem: Jump, V.92, Version 1.0 Bluetooth: Version 2.1.3f8, 2 service, 0 devices, 1 incoming serial ports Network Service: AirPort, AirPort, en1 Parallel ATA Device: MATSHITACD-RW CW-8123 Parallel ATA Device: ST980815A, 74,53 GB USB Device: Bluetooth USB Host Controller, (null) mA We checked again that the condition variable is correctly allocated with this kind of code: This email sent to site_archiver(a)lists.apple.com

1 0