December 2006 - Darwin-dev - lists.apple.com

Re: further help on debugging a hang
by Giuliano Gavazzi 13 Dec '06

13 Dec '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com http://www.humph.com/tmp/dump.txt it's about 350KB 6536 lines. I have sent this same file to ACARD for them to examine. Giuliano _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Thanks to all for the advice. I knew of the Kernel Debug Kit, my main problem was to be sure that the correct key to press for an NMI was the recessed and not the prominent one (the first hit I found was indeed from someone who was thinking from inside the computer, and was, incorrectly for us outside, defining the NMI as the convex one...). Unfortunately a hang means that I have too much information and not a nice kernel panic that usually happens near the source of trouble. If anyone accustomed to kernel dump's showallstacks would like to have a look at it: Thanks again and if there is anything else you think I should do let me know. This email sent to site_archiver(a)lists.apple.com

1 0

Re: further help on debugging a hang
by Garth Cummings 13 Dec '06

13 Dec '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Re: further help on debugging a hang
by Brian Bechtel 13 Dec '06

13 Dec '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=cNwRVqoR0gzTkgisY0Cv/G2q/dXL2pbLt1hzEop5XjWU3XAVrGrtwg03rIq0aKbdSJjjsL8htIPDk3YH4Hj10GILG3+9BxnGTXhzLJS4g+1DE4oUXYPMnRu3rgJzDu7X/OGKojoX/JrhVYt5Dbq+NfI1p+oBEoskR9f9WjlvbP4= minor correction which I noticed as soon as I sent the first email: -a com.apple.iokit.IOSCSIFamily@0x234567 \ -a com.acard.driver.ACard68xxM@0xADDR_FROM_SHOWALLKMODS \ -s ~/Desktop/gg -- /path/to/ACard68xxM_driver.kext every "-a" argument is of the form drivername@address "-s" defines the directory to store the symbol files The path to the driver is typically /System/Library/Extensions/ACard68xMM.kext that would replace "/path/to/ACard68xxM_driver.kext" Again, good luck _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On 12/12/06, Brian Bechtel <brian.bechtel(a)gmail.com> wrote: kextload -a com.apple.iokit.IOPCIFamily@0x123456 \ This email sent to site_archiver(a)lists.apple.com

1 0

Re: further help on debugging a hang
by Brian Bechtel 13 Dec '06

13 Dec '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=VulA9Tds3Uo/+sJYNBPu++N25/2043t/mJxtw/mouUFnTHQUGXcdOFHUXSjIn808xke1/XFRqyxXjiBe22AvhKBIYXpI0N7C5JvEsKJOcpMfvBelLBYnQuxMeu3FtnLlpAS+DUZGcll/62WdhORnrL8ZnPAFcpgvJvwwhMhOAAE= When it hangs, NMI the machine to dump core. This assumes you have the boot-args set as sudo nvram boot-args="debug=0xd44" or equivalent, as documented in Apple's technical note. Once core is dumped, reboot the machine. You can tell when the coredump is finshed because you will see the message "waiting for some to attach" or equivalent. (see support.apple.com and search for "NMI" to see what the NMI key combinations for your machine are.) Now that you have the core file, it will be named something like core-xnu-792-XX.XX.XX.XX-xxxxxxxxxx where the XX.XX.XX.XX.XX is the IP address of the machine you are trying to debug, and the other x's are hexadecimal. Download the kernel debug kit for 10.4.8 from developer.apple.com/sdk. Open gdb and pass it the "-c" option, using the mach_kernel from 10.4.8 as the other argument. gdb -c core-xnu-792-XX.XX.XX.XX-xxxxxxxxxx mach_kernel (gdb) source kgmacros (gdb) showallstacks look for stacks which have I/O pending (you should see something related to the ACARD kext if your theory is correct.) You can add symbols for the ACARD kext by the following complicated process: (gdb) showallkmods find the ACARD kext, look in the second column for the address where it is loaded. (I downloaded the latest driver from http://dl.acard.com/download/mac/ACARD_Driver_1.5.7.pkg.sit) to get the following information. The driver is com.acard.driver.ACard68xxM In a different terminal window, enter the command mkdir ~/Desktop/gg kextload -a com.apple.iokit.IOPCIFamily@0x123456 -a com.apple.iokit.IOSCSIFamily@0x234567 -a com.acard.driver.ACard68xxM -s ~/Desktop/gg -- /path/to/ACard68xxM_driver.kext this will generate the symbol files. Now in gdb, add them with (gdb) add-symbol-file ~/Desktop/gg/com.acard.driver.ACard68xxM.sym and entering "yes" when prompted. Now, you do the hard part. You debug. You debug. You debug. Good luck. Remember, all of this is from memory, not from trying it myself. Your experience may vary. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On 12/12/06, Giuliano Gavazzi <dev+lists(a)humph.com> wrote: What course of action would you take to debug it? This email sent to site_archiver(a)lists.apple.com

1 0

Network Bandwidth
by Anees Alappat 13 Dec '06

13 Dec '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:mime-version:content-type; b=h76OX8poVTHq+qc5CcXvhsrcVW9Pynucn6Lzrv3qp5Y3KTbVRAwvb69koE/dkV+FsNwJj6hN4wd6ZAimmHrouHwA0tZK3r6ezt35CvnQH2yXDL0Aun/lGM3BoZlLG4AEAIulTRYLHVOjeopDyMk9e95DliCDg0iWo6z6MbYDn5U= _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: further help on debugging a hang
by Dan Shoop 13 Dec '06

13 Dec '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com A recap: Likely What happens when you remove it and the driver from the system? Doubt it's lookupd -- -dhan ------------------------------------------------------------------------ Dan Shoop AIM: iWiring Systems & Networks Architect http://www.ustsvs.com/ shoop(a)iwiring.net http://www.iwiring.net/ 1-714-363-1174 "The wise man doesn't give the right answers, he poses the right questions." -- Claude Levi-Strauss ------------------------------------------------------------------------ iWiring provides systems and networks support for Mac OS X, unix, and Open Source application technologies at affordable rates. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… At 1:57 AM +0100 12/13/06, Giuliano Gavazzi wrote: sorry to disturb again, but the machine I am debugging cannot be left off-line too long as it is a production server (and is at a remote location), so I am looking for some advice. PowerMacG4 DP400 10.4.8 + security update 007 + ACARD 6890M SATA RAID configured for mirroring. Symptoms: every few days the machine hangs (it looks like the old "rainbow of death") remaining responsive to pings but otherwise inaccessible from both the console and ssh. Even telnet to an open tcp port (say 80) will establish the connection but the process on the other side will not respond any further (in this example a GET / HTTP/1.0 <cr> <cr> will not give anything). Right, because it's hung in an [disk] IO wait state that needs to complete before any further IO can occur. All of the above require IO so you're blocked while the wait ccompletes, I have two prime suspects: the ACARD (driver or card) and lookupd. This email sent to site_archiver(a)lists.apple.com

1 0

Re: further help on debugging a hang
by Chris Zubrzycki 13 Dec '06

13 Dec '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Dec 12, 2006, at 7:57 PM, Giuliano Gavazzi wrote: A recap: - -chris zubrzycki - - -- PGP public key: http://homepage.mac.com/beren/publickey.txt ID: 0xA2ABC070 Fingerprint: 26B0 BA6B A409 FA83 42B3 1688 FBF9 8232 A2AB C070 ======================================================== "One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie." -- The Silicon Valley Tarot -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) Comment: Please sign reply-http://www.gnupg.org iEYEARECAAYFAkV/VA8ACgkQ+/mCMqKrwHDaSQCgx97a7tUTO8n8N2pWsJWgAEcO xjkAn0U7QNh+i+TuYRZZliIe/H1QK48l =/rLX -----END PGP SIGNATURE----- _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… sorry to disturb again, but the machine I am debugging cannot be left off-line too long as it is a production server (and is at a remote location), so I am looking for some advice. PowerMacG4 DP400 10.4.8 + security update 007 + ACARD 6890M SATA RAID configured for mirroring. Symptoms: every few days the machine hangs (it looks like the old "rainbow of death") remaining responsive to pings but otherwise inaccessible from both the console and ssh. Even telnet to an open tcp port (say 80) will establish the connection but the process on the other side will not respond any further (in this example a GET / HTTP/1.0 <cr> <cr> will not give anything). I have two prime suspects: the ACARD (driver or card) and lookupd. The ACARD is suspect because this is the only machine running 10.4.8 I have mounting this card and the only one suffering from these hangs. lookupd is suspect simply because the symptoms are so similar to what one would get when lookupd stops working. I would try to rule out lookupd. Maybe have it log the most verbose (but it'll spam syslog too unless you block it) and log it to a separate file, to examine after a hang. man lookupd will show the way. Also, you could kill and restart lookupd maybe every 24 hours or so. That might be enough to keep it alive, if indeed it is lookupd dying. Those are some troubleshooting methods I would use. This email sent to site_archiver(a)lists.apple.com

1 0

further help on debugging a hang
by Giuliano Gavazzi 13 Dec '06

13 Dec '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com A recap: What course of action would you take to debug it? Thank you Giuliano _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… sorry to disturb again, but the machine I am debugging cannot be left off-line too long as it is a production server (and is at a remote location), so I am looking for some advice. PowerMacG4 DP400 10.4.8 + security update 007 + ACARD 6890M SATA RAID configured for mirroring. Symptoms: every few days the machine hangs (it looks like the old "rainbow of death") remaining responsive to pings but otherwise inaccessible from both the console and ssh. Even telnet to an open tcp port (say 80) will establish the connection but the process on the other side will not respond any further (in this example a GET / HTTP/1.0 <cr> <cr> will not give anything). I have two prime suspects: the ACARD (driver or card) and lookupd. The ACARD is suspect because this is the only machine running 10.4.8 I have mounting this card and the only one suffering from these hangs. lookupd is suspect simply because the symptoms are so similar to what one would get when lookupd stops working. The machine, as I wrote previously, is set now to perform a kernel dump to another machine on the LAN, but I am not sure at all that this will help. Too much seems to be running still for the dump to point at the source of the problem. I hope to be proven wrong when I will manage to get the kernel dump. This email sent to site_archiver(a)lists.apple.com

1 0

Debugging a hang Was: Problem with Technical Note...
by Giuliano Gavazzi 12 Dec '06

12 Dec '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Thanks in advance Giuliano _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I finally have a hang from the machine I am observing (and setup according to TN2118). It is a PowerMac G4 DP400MHz. Since I am not on location I will give instructions to press the NMI button to cause the kernel dump to the kdumpd server. Just to be sure, the NMI button is the convex one that one "usually" presses on a PMac G4 to restart it, isn't it!? I ask as one method listed in Kernel Programming, namely "command, option, control, and shift keys and hit escape" does not work on a bluetooth keyboard (just tested on the mini at my side). Perhaps it would work on a usb keyboard (and I might try on the one I am writing on..). To the curious this hangs might be related to the recent security update and the installed ACARD 6890M SATA RAID PCI card. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Preventing system sleep, QA1340 code does not seem to work
by Ethan Bold 12 Dec '06

12 Dec '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Hi Sean- Yep. I'm stumped. -Ethan On Dec 12, 2006, at 1:42 PM, Sean McBride wrote: Hi all, QA1340 "Registering and unregistering for sleep notifications" is just what I was looking for. <http://developer.apple.com/qa/qa2004/qa1340.html> I copy and pasted the code from 'listing 1' and put it in a brand new Xcode 'CoreFoundation tool' project. But the callback function specified to IORegisterForSystemPower() is never called! I've put a breakpoint, printf, and assert(0). Thanks, -- ____________________________________________________________ Sean McBride, B. Eng sean(a)rogue-research.com Rogue Research www.rogue-research.com Mac Software Developer Montréal, Québec, Canada _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/ebold%40apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… To be clear, the kIOMessageCanSystemSleep message only allows you to prevent IDLE sleep. If you close the clamshell or sleep via the Apple menu, your callback will never receive the kIOMessageCanSystemSleep message. There is no way to prevent non-idle sleeps. That said, yes, this code should work in 10.3.9, 10.4.8, and 9A303. I use it all the time. * Are you getting any error returns from any of the IOKit or CF calls in main()? * Where are your breaks/printfs/asserts - are you breaking on the invocation of your callback function itself? Or only in the kIOMessageCanSystemSleep code (which will not get called on a force sleep)? Is anyone using this code? I've tried in 10.3.9, 10.4.8, and 9a303 and it just doesn't seem to work at all. (PS: yes, I have a good reason to deny sleep: firmware update over RS-232.) This email sent to ebold(a)apple.com This email sent to site_archiver(a)lists.apple.com

1 0