July 2008 - Darwin-dev - lists.apple.com

Re: getaddrinfo() starts failing with EAI_AGAIN (again)
by Terry Lambert 10 Jul '08

10 Jul '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Personally, I'd be looking for horses, not zebras, at this point. -- Terry On Jul 10, 2008, at 5:36 AM, Jamus Jegier <jamus(a)jamus.com> wrote: On Jul 9, 2008, at 10:21 PM, Terry Lambert wrote: #!/bin/bash whoami >> /tmp/debug.log date >> /tmp/debug.log dig $1 >> /tmp/debug.log ping -c 5 $1>> /tmp/debug.log _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… It is probably not being "fixed" by that for the reasons being assumed by that explanation. The person I would ask for a definative answer on whether it was as Peter suggests is on vacation. You said this was a long running process. The daemon that answers lookup information registers in the system lookupd bootstrap namespace, not the "session" bootstrap namespace, so it's somewhat doubtful to me that losing access to the latter would/should impact you this way. We internally use exactly this approach for distributed builds, and it seems to me they would break on us the same way for the same reasons, if this were the case. It also seems to me the latency would be a heck of a lot less than 8 hours, if this were a "session" based port rights revocation issue; more like "the instant the session ends". An outside possibility is that some degenerate behaviour on this code path is causing Mach ports to leak like there's no tomorrow. There are tools to check for this as well, but you should be seeing much worse behavior than just one app failing, were that the case. On Jul 7, 2008, at 8:41 AM, Jamus Jegier wrote: A couple months ago, there was a post by Peter Oberauer about getaddrinfo() failing with EAI_AGAIN by processes under a specific process tree. I am running into the exact same problem, and was wondering if there was any resolution to the issue. I installed Nagios, which frequently forks processes to verify the status of network hosts. After about 8 hours, all child processes start failing with EAI_AGAIN. Under the Nagios parent process, dig works, but ping doesn't. Both DNS and Bonjour lookups begin to fail. Ping and the Nagios helper executables work fine when started under an unrelated process. The only other thing I can add is that I see this on 10.5.3 on a G4 system. I'm installing 10.5.4 now, and will post if I still see this. Typically, getaddrinfo() fails with EAI_AGAIN when there is a failure of malloc of the memory needed to return the linked list of addrinfo structures. This generally happens if: (1) you have a memory leak (either you fail to call freeaddrinfo() on the returned memory, or have a different leak) (2) you have fragmented your process address space (sufficiently that it is impossible to allocate a contiguous memory chunk large enough to return the requested information_ (3) the information you are requesting would be so large that it's impossible to return it (misconfigured DNS server, DNS cache poisoning attack, broken DNS server software, etc.) You should examine how much memory is in use by your process to distinguish #1, you should use vmmap to distinguish #2, and you should use host and similar commands and/or a packet analyzer (or tcpdump) to distinguish #3. Thanks for the suggestions, but I believe Peter Oberauer hit the nail on the head - starting the process from launchd solved my issues. Before that nagios was started by sshing into my system and sudo to the nagios user like this: sudo -u nagios /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/ nagios.cfg Nagios would run this script every few minutes successfully before it would start failing after approximately 8-10 hours. In this script, ping would fail while dig would succeed in resolving the domain name. So I don't think it would be #2, seeing that I doubt that ping would allocate and fragment its address space like you suggested. Also, I don't think it's #3, since bonjour also fails with a directly connected Airport Extreme, dig returns a good value with DNS queries, and pings outside of the nagios process tree would succeed I also don't think it's #1, since I would have noticed ping thrashing as it tried to allocate 4gb in my 1.5 gb system. I can attempt to reproduce the issue and try to gather hard data if my observations aren't sufficient. Jamus This email sent to site_archiver(a)lists.apple.com

1 0

Re: getaddrinfo() starts failing with EAI_AGAIN (again)
by Jamus Jegier 10 Jul '08

10 Jul '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=simple/simple; d=jamus.com; s=dkim; t=1215693420; bh=FWqf6AhGxQrcQn5J4SN+k9eAVMWooFqkdGZBMEoa/yc=; h=Cc:Message-Id:From:To:In-Reply-To:Content-Type: Content-Transfer-Encoding:Mime-Version:Subject:Date:References; b=H+ER6Pb6/66DE57k/38JKkeOxxO8CxSS67ZhA3TPm6Q+7sKPJl+X7hDWjn6vgBpma suFxZ8urH6UprJWqKKHtYcfTuHeE6dALeMwySLpLNE+4TGYg7eci4ORHUPQASGtWwfo p9MKQM8loPPq3RYq7LJZwt53WY2pyctsnLOMops= Domainkey-signature: a=rsa-sha1; s=dk; d=jamus.com; c=simple; q=dns; h=cc:message-id:from:to:in-reply-to:content-type: content-transfer-encoding:mime-version:subject:date:references:x-mailer; b=s7nnhXJtiqOC+v4FayXnqoHMFOKhM2tmHaKI22gikE1/ASMLg01DCYrHkc/dCRIY9 5k4OQBLhBy9znueynM13A== On Jul 9, 2008, at 10:21 PM, Terry Lambert wrote: #!/bin/bash whoami >> /tmp/debug.log date >> /tmp/debug.log dig $1 >> /tmp/debug.log ping -c 5 $1>> /tmp/debug.log Jamus _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Jul 7, 2008, at 8:41 AM, Jamus Jegier wrote: A couple months ago, there was a post by Peter Oberauer about getaddrinfo() failing with EAI_AGAIN by processes under a specific process tree. I am running into the exact same problem, and was wondering if there was any resolution to the issue. I installed Nagios, which frequently forks processes to verify the status of network hosts. After about 8 hours, all child processes start failing with EAI_AGAIN. Under the Nagios parent process, dig works, but ping doesn't. Both DNS and Bonjour lookups begin to fail. Ping and the Nagios helper executables work fine when started under an unrelated process. The only other thing I can add is that I see this on 10.5.3 on a G4 system. I'm installing 10.5.4 now, and will post if I still see this. Typically, getaddrinfo() fails with EAI_AGAIN when there is a failure of malloc of the memory needed to return the linked list of addrinfo structures. This generally happens if: (1) you have a memory leak (either you fail to call freeaddrinfo() on the returned memory, or have a different leak) (2) you have fragmented your process address space (sufficiently that it is impossible to allocate a contiguous memory chunk large enough to return the requested information_ (3) the information you are requesting would be so large that it's impossible to return it (misconfigured DNS server, DNS cache poisoning attack, broken DNS server software, etc.) You should examine how much memory is in use by your process to distinguish #1, you should use vmmap to distinguish #2, and you should use host and similar commands and/or a packet analyzer (or tcpdump) to distinguish #3. Thanks for the suggestions, but I believe Peter Oberauer hit the nail on the head - starting the process from launchd solved my issues. Before that nagios was started by sshing into my system and sudo to the nagios user like this: sudo -u nagios /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/ nagios.cfg Nagios would run this script every few minutes successfully before it would start failing after approximately 8-10 hours. In this script, ping would fail while dig would succeed in resolving the domain name. So I don't think it would be #2, seeing that I doubt that ping would allocate and fragment its address space like you suggested. Also, I don't think it's #3, since bonjour also fails with a directly connected Airport Extreme, dig returns a good value with DNS queries, and pings outside of the nagios process tree would succeed I also don't think it's #1, since I would have noticed ping thrashing as it tried to allocate 4gb in my 1.5 gb system. I can attempt to reproduce the issue and try to gather hard data if my observations aren't sufficient. This email sent to site_archiver(a)lists.apple.com

1 0

Compiling hfsp as vfs module
by Bart Thissen 10 Jul '08

10 Jul '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Organization: ArdisTechnologies bv User-agent: Mozilla-Thunderbird 2.0.0.9 (X11/20080110) Hi, Cheers, Bart _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… For debugging/tracing purposes I'd like to modify the HFSP sources out of the xnu/bsd/hfs subdirectory such that I can compile it as a VFS module (of course with a different fs name chosen), and kextload/unload it. Did somebody do this already and is this difficult to do? This email sent to site_archiver(a)lists.apple.com

1 0

Re: getaddrinfo() starts failing with EAI_AGAIN (again)
by Terry Lambert 10 Jul '08

10 Jul '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Jul 7, 2008, at 8:41 AM, Jamus Jegier wrote: A couple months ago, there was a post by Peter Oberauer about getaddrinfo() failing with EAI_AGAIN by processes under a specific process tree. I am running into the exact same problem, and was wondering if there was any resolution to the issue. I installed Nagios, which frequently forks processes to verify the status of network hosts. After about 8 hours, all child processes start failing with EAI_AGAIN. Under the Nagios parent process, dig works, but ping doesn't. Both DNS and Bonjour lookups begin to fail. Ping and the Nagios helper executables work fine when started under an unrelated process. The only other thing I can add is that I see this on 10.5.3 on a G4 system. I'm installing 10.5.4 now, and will post if I still see this. Typically, getaddrinfo() fails with EAI_AGAIN when there is a failure of malloc of the memory needed to return the linked list of addrinfo structures. This generally happens if: (1) you have a memory leak (either you fail to call freeaddrinfo() on the returned memory, or have a different leak) (2) you have fragmented your process address space (sufficiently that it is impossible to allocate a contiguous memory chunk large enough to return the requested information_ (3) the information you are requesting would be so large that it's impossible to return it (misconfigured DNS server, DNS cache poisoning attack, broken DNS server software, etc.) You should examine how much memory is in use by your process to distinguish #1, you should use vmmap to distinguish #2, and you should use host and similar commands and/or a packet analyzer (or tcpdump) to distinguish #3. This email sent to site_archiver(a)lists.apple.com

1 0

Re: IOUserClient changed since 10.3? Can't load kext on 10.3.9
by Boris Remizov 09 Jul '08

09 Jul '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:cc:message-id:from:to :in-reply-to:content-type:content-transfer-encoding:mime-version :subject:date:references:x-mailer; bh=8Oi90T3XmmFyj0ORCQa3teR17JszjQ0L443HWFDGfI8=; b=wPkpCMTXzkWN+7LGXAGibS5JiskqXc/fJ6fsBrnZ1Yermx6mFxzIbQP3pNHMe/GakN iUDVYIwxIb1CgI1Ok9LlRRNfbqIN3EECMWBgGye0XJoLePgAttsDgOipa0D8jHB4v4NI haGTQk1EQFzF75ZKW+da5jon5RTFFM2Am9S5Y= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=cc:message-id:from:to:in-reply-to:content-type :content-transfer-encoding:mime-version:subject:date:references :x-mailer; b=dc6Qs6zeQMVD0xtl/VWSR88B0uXKB7t4sOv8hypRYenHvldnz6Ky64g7WeOS04gbpp yualBXDwwCDtUnJwe9JPy2kfbDpjfw8ZPCasxLV+L/lByw49fIMZWZigsMD6gGQWZCpk Gz6e8DP9iTFY4Hw/NtWaVCuHGxWcBkxiShETk= On Jul 9, 2008, at 2:37 PM, Andy Green wrote: I've got a kext that loads successfully on 10.4 onwards. However, it won't load on 10.3.9: kextload reports *** is not compatible with it's superclass, 12IOUserClient superclass changed? It looks like someone else had this problem - see http://lists.apple.com/archives/darwin-drivers/2006/Feb/msg00017.html Unfortunately the solution didn't make it onto the mailing list!! Can anyone enlighten me? Some of plist copied below:- [Incidentally, I was seeing other issues from kextload, saying that:- "Kext has a kernel dependency prior to version 6.0 and CFBundleVersion does not match executable's MODULE_VERSION" = true I've fixed it by declaring a dependency on iokit/libkern v6.9.9 (MacOS 10.3). Not sure it really mattered, but I'd like to understand it!/ <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>CFBundleDevelopmentRegion</key> <string>English</string> <key>CFBundleExecutable</key> <string>${EXECUTABLE_NAME}</string> <key>CFBundleName</key> <string>${PRODUCT_NAME}</string> <key>CFBundleIconFile</key> <string></string> <key>CFBundleIdentifier</key> <string> ***** </string> <key>CFBundleInfoDictionaryVersion</key> <string>6.0</string> <key>CFBundlePackageType</key> <string>KEXT</string> <key>CFBundleSignature</key> <string>????</string> <key>CFBundleVersion</key> <string>0.09.2</string> <key>OSBundleCompatibleVersion</key> <string>0.09.0</string> <key>IOKitPersonalities</key> <dict> <key>****</key> <dict> <key>CFBundleIdentifier</key> <string>****</string> <key>IOClass</key> <string>****</string> <key>IOProviderClass</key> <string>IOSCSIPeripheralDeviceNub</string> <key>Peripheral Device Type</key> <integer>0</integer> **** </dict> </dict> <key>OSBundleLibraries</key> <dict> <key>com.apple.iokit.IOSCSIArchitectureModelFamily</key> <string>1.0.0</string> <key>com.apple.iokit.IOSCSIBlockCommandsDevice</key> <string>1.0.0</string> <key>com.apple.iokit.IOStorageFamily</key> <string>1.0.0</string> <key>com.apple.kernel.iokit</key> <string>6.9.9</string> <key>com.apple.kernel.libkern</key> <string>6.9.9</string> </dict> <key>OSBundleRequired</key> <string>Local-Root</string> </dict> </plist> _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Hello, I think you must build your kext with appropriate SDK (MacOSX 10.3.9 in this case). This email sent to site_archiver(a)lists.apple.com

1 0

IOUserClient changed since 10.3? Can't load kext on 10.3.9
by Andy Green 09 Jul '08

09 Jul '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Thread-index: AcjhLWPbxi6fZBlXSCaCb9CAt1cfPgAgDYng Thread-topic: IOUserClient changed since 10.3? Can't load kext on 10.3.9 I've got a kext that loads successfully on 10.4 onwards. However, it won't load on 10.3.9: kextload reports *** is not compatible with it's superclass, 12IOUserClient superclass changed? It looks like someone else had this problem - see http://lists.apple.com/archives/darwin-drivers/2006/Feb/msg00017.html Unfortunately the solution didn't make it onto the mailing list!! Can anyone enlighten me? Some of plist copied below:- [Incidentally, I was seeing other issues from kextload, saying that:- "Kext has a kernel dependency prior to version 6.0 and CFBundleVersion does not match executable's MODULE_VERSION" = true I've fixed it by declaring a dependency on iokit/libkern v6.9.9 (MacOS 10.3). Not sure it really mattered, but I'd like to understand it!/ <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>CFBundleDevelopmentRegion</key> <string>English</string> <key>CFBundleExecutable</key> <string>${EXECUTABLE_NAME}</string> <key>CFBundleName</key> <string>${PRODUCT_NAME}</string> <key>CFBundleIconFile</key> <string></string> <key>CFBundleIdentifier</key> <string> ***** </string> <key>CFBundleInfoDictionaryVersion</key> <string>6.0</string> <key>CFBundlePackageType</key> <string>KEXT</string> <key>CFBundleSignature</key> <string>????</string> <key>CFBundleVersion</key> <string>0.09.2</string> <key>OSBundleCompatibleVersion</key> <string>0.09.0</string> <key>IOKitPersonalities</key> <dict> <key>****</key> <dict> <key>CFBundleIdentifier</key> <string>****</string> <key>IOClass</key> <string>****</string> <key>IOProviderClass</key> <string>IOSCSIPeripheralDeviceNub</string> <key>Peripheral Device Type</key> <integer>0</integer> **** </dict> </dict> <key>OSBundleLibraries</key> <dict> <key>com.apple.iokit.IOSCSIArchitectureModelFamily</key> <string>1.0.0</string> <key>com.apple.iokit.IOSCSIBlockCommandsDevice</key> <string>1.0.0</string> <key>com.apple.iokit.IOStorageFamily</key> <string>1.0.0</string> <key>com.apple.kernel.iokit</key> <string>6.9.9</string> <key>com.apple.kernel.libkern</key> <string>6.9.9</string> </dict> <key>OSBundleRequired</key> <string>Local-Root</string> </dict> </plist> _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

IOUserClient changed since 10.3? Can't load kext on 10.3.9
by Andy Green 09 Jul '08

09 Jul '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Thread-index: AcjhLWPbxi6fZBlXSCaCb9CAt1cfPgAgDYngAABQi2A= Thread-topic: IOUserClient changed since 10.3? Can't load kext on 10.3.9 I've got a kext that loads successfully on 10.4 onwards. However, it won't load on 10.3.9: kextload -t reports : *** is not compatible with it's superclass, 12IOUserClient superclass changed? kextload -nt says "appears to be valid", but actual load won't work It looks like someone else had this problem - see http://lists.apple.com/archives/darwin-drivers/2006/Feb/msg00017.html Unfortunately the solution didn't make it onto the mailing list!! Can anyone enlighten me? Some of plist copied below:- [Incidentally, I was seeing other issues from kextload, saying that:- "Kext has a kernel dependency prior to version 6.0 and CFBundleVersion does not match executable's MODULE_VERSION" = true I've fixed it by declaring a dependency on iokit/libkern v6.9.9 (MacOS 10.3). Not sure it really mattered, but I'd like to understand it!/ ----------------- <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>CFBundleDevelopmentRegion</key> <string>English</string> <key>CFBundleExecutable</key> <string>${EXECUTABLE_NAME}</string> <key>CFBundleName</key> <string>${PRODUCT_NAME}</string> <key>CFBundleIconFile</key> <string></string> <key>CFBundleIdentifier</key> <string> ***** </string> <key>CFBundleInfoDictionaryVersion</key> <string>6.0</string> <key>CFBundlePackageType</key> <string>KEXT</string> <key>CFBundleSignature</key> <string>????</string> <key>CFBundleVersion</key> <string>0.09.2</string> <key>OSBundleCompatibleVersion</key> <string>0.09.0</string> <key>IOKitPersonalities</key> <dict> <key>****</key> <dict> <key>CFBundleIdentifier</key> <string>****</string> <key>IOClass</key> <string>****</string> <key>IOProviderClass</key> <string>IOSCSIPeripheralDeviceNub</string> <key>Peripheral Device Type</key> <integer>0</integer> **** </dict> </dict> <key>OSBundleLibraries</key> <dict> <key>com.apple.iokit.IOSCSIArchitectureModelFamily</key> <string>1.0.0</string> <key>com.apple.iokit.IOSCSIBlockCommandsDevice</key> <string>1.0.0</string> <key>com.apple.iokit.IOStorageFamily</key> <string>1.0.0</string> <key>com.apple.kernel.iokit</key> <string>6.9.9</string> <key>com.apple.kernel.libkern</key> <string>6.9.9</string> </dict> <key>OSBundleRequired</key> <string>Local-Root</string> </dict> </plist> _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: getaddrinfo() starts failing with EAI_AGAIN (again) (Jamus Jegier)
by Peter Oberauer 08 Jul '08

08 Jul '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: getaddrinfo() starts failing with EAI_AGAIN (again) (Jamus Jegier)
by Jamus Jegier 07 Jul '08

07 Jul '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=simple/simple; d=jamus.com; s=dkim; t=1215473583; bh=QmniJqIyp02fO1ZBNX+1snjRq8zhA3RYUhbbOTp5oB4=; h=Cc:Message-Id:From:To:In-Reply-To:Content-Type:Mime-Version: Subject:Date:References; b=THggkJ+PqwaF2Vh0PvrA+JMMOFz09AxhdM8yrU5 IavPdmnAQTdJ/kr5GQ4Ud/HfAbkDgCu6Oz+gNjrBTrGJYEaTM6MTv4FO/dSfaRpbugA NuaIvERp8C7ox7TjXmvhzs9OYQBf+iWjtx+zXJsM1ozQCBOKss9YWdw/Tszmvik1Q= Domainkey-signature: a=rsa-sha1; s=dk; d=jamus.com; c=simple; q=dns; h=cc:message-id:from:to:in-reply-to:content-type: mime-version:subject:date:references:x-mailer; b=qm45Lm4p1zgBa5Hs2YC35sy3o5eYJfouZQQG+EYlagN7/+0WrvRSB3QgI6MUFmkKA Q0AICXCKCPZIT8IJogbpg== _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: getaddrinfo() starts failing with EAI_AGAIN (again) (Jamus Jegier)
by Peter Oberauer 07 Jul '08

07 Jul '08

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0