December 2006 - Darwin-kernel

Re: utimes() no good for symbolic links?
by Jerry Krinock 16 Dec '06

16 Dec '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Thread-index: Accg45Vt1D3EZozWEducvAAWy6NyrAASpLGJ Thread-topic: utimes() no good for symbolic links? User-agent: Microsoft-Entourage/11.2.5.060620 OK, I tried FSSetCatalogInfo, and it fails for symlinks also, a little differently. FSPathMakeRef seems to be trying to follow the link, and instead of setting the times to the current times like utimes() does, it leaves the times unchanged. The Finder does not copy "Date Modified" as expected either. If I File > Duplicate a regular file, the "Date Modified" of the copy is the same as that of the original. But if I File > Duplicate a symlink, the the "Date Modified" of the copy is the current time. I'm getting a little off topic with the Carbon stuff, but if there is a solution to this problem, I suspect it is at the BSD layer. Any ideas? Thanks, Jerry Krinock ********* TEST PROGRAM FOR FSSetCatalogInfo ************ Sorry for the kludgey conversion of seconds to years for display, but the only way I know to do that uses Cocoa (NSDate), and I wanted to keep this easily reproducible for the unix folks. #include <stdio.h> #import <CoreServices/CoreServices.h> #include <sys/stat.h> #define SECS_PER_YEAR (365.25 * 24 * 60 * 60) int main (int argc, const char * argv[]) { UInt32 y2k2_secs = (2002.2 - 1904) * SECS_PER_YEAR ; UInt32 y2k3_secs = (2003.2 - 1904) * SECS_PER_YEAR ; UTCDateTime y2k2_utc = {0, y2k2_secs, 0} ; UTCDateTime y2k3_utc = {0, y2k3_secs, 0} ; // Verify that the system thinks it can set this info: printf(" kFSCatInfoSettableInfo = %4x\n", kFSCatInfoSettableInfo) ; printf(" access time settable? %4x\n", kFSCatInfoSettableInfo & kFSCatInfoAccessDate) ; printf("modification time settable? %4x\n", kFSCatInfoSettableInfo & kFSCatInfoContentMod) ; // Make FSRef FSRef pathRef; OSErr osErr = FSPathMakeRef((UInt8*)"/f1", &pathRef, NULL); printf("osErr %i from FSPathMakeRef\n", osErr) ; // Change file to have these atime and mtime FSCatalogInfo catInfo ; catInfo.accessDate = y2k2_utc ; catInfo.contentModDate = y2k3_utc ; osErr = FSSetCatalogInfo( &pathRef, kFSCatInfoAccessDate | kFSCatInfoContentMod, &catInfo) ; printf("osErr %i from FSSetCatalogInfo\n", osErr) ; // Now read the times back. Use lstat because gives info about // the symlink, not the symlink target, for symlinks struct stat stats ; lstat("/f1", &stats) ; printf("f1 atime: %f\n", stats.st_atime/SECS_PER_YEAR + 1970) ; printf("f1 mtime: %f\n", stats.st_mtime/SECS_PER_YEAR + 1970) ; return 0; } ********* RESULTS ********************** SETUP. Since man utimes (2) says that you must be the super user, compile the program and run under sudo, just to make sure. Create a file at path /f1. First of all, my system claims that both times are settable: kFSCatInfoSettableInfo = 1fe3 access time settable? 100 modification time settable? 40 Case 1. If /f1 is a regular file, I get the desired result, with the times set to sometime in 2002 and 2003: osErr 0 from FSPathMakeRef osErr 0 from FSSetCatalogInfo f1 atime: 2002.198631 f1 mtime: 2003.198631 Case 2. If /f1 is a valid symlink, functions return no error but the times are unchanged: osErr 0 from FSPathMakeRef osErr 0 from FSSetCatalogInfo f1 atime: 2006.956309 f1 mtime: 2006.956309 Case 3. If /f1 is a broken symlink, FSPathMakeRef returns "file not found" error and the times are unchanged: osErr -43 from FSPathMakeRef osErr -35 from FSSetCatalogInfo f1 atime: 2006.956309 f1 mtime: 2006.956309 _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

utimes() no good for symbolic links?
by Jerry Krinock 16 Dec '06

16 Dec '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Thread-index: Accg45Vt1D3EZozWEducvAAWy6NyrA== Thread-topic: utimes() no good for symbolic links? User-agent: Microsoft-Entourage/11.2.5.060620 It looks to me like utimes() does not work for symbolic links. If the symlink is valid, it returns 0, but sets both atime and mtime to the current time instead of the time I pass in as argument. If the symlink is broken, it returns -1 and also sets atime and mtime to the current time. The documentation man utimes (2) does not mention that there is any problem with symbolic links. Does anyone know, is this a bug in utimes(), a documentation bug, or my misunderstanding? Also, does anyone know of a good function that works for setting atime and mtime of all files, including symlinks? I don't think that FSSetCatalogInfo works for symlinks either, but it's too late to try tonight. And even if it does, it ain't documented. Thanks, Jerry Krinock ********* TEST PROGRAM ************ Sorry for the kludgey conversion of seconds to years for display, but the only way I know to do that uses Cocoa (NSDate), and I wanted to keep this easily reproducible for the unix folks. #include <stdio.h> #import <CoreServices/CoreServices.h> #include <sys/stat.h> #define SECS_PER_YEAR (365.25 * 24 * 60 * 60) int main (int argc, const char * argv[]) { __darwin_time_t y2k2_secs = (2002.2 - 1970) * SECS_PER_YEAR ; __darwin_time_t y2k3_secs = (2003.2 - 1970) * SECS_PER_YEAR ; struct timeval tval[2] ; // Set atime to some time in 2002. tval[0].tv_sec = y2k2_secs ; tval[0].tv_usec = 0; // Set mtime to some time in 2003 tval[1].tv_sec = y2k3_secs ; tval[1].tv_usec = 0; // Change file to have this atime and mtime int err ; err = utimes("/f1", tval) ; printf("err utimes = %i\n", err) ; // Now read the times back. Use lstat because gives info about // the symlink, not the symlink target, for symlinks struct stat stats ; err = lstat("/f1", &stats) ; printf("f1 atime: %f\n", stats.st_atime/SECS_PER_YEAR + 1970) ; printf("f1 mtime: %f\n", stats.st_mtime/SECS_PER_YEAR + 1970) ; return 0; } ********* RESULTS ********************** SETUP. Since man utimes (2) says that you must be the super user, compile the program and run under sudo. Create a file at path /f1. Case 1. If /f1 is a regular file, I get the desired result, with the times set to sometime in 2002 and 2003: err utimes = 0 f1 atime: 2002.200000 f1 mtime: 2003.200000 Case 2. If /f1 is a valid symlink, utimes() returns no error but the times are incorrectly set to the current times: err utimes = 0 f1 atime: 2006.956309 f1 mtime: 2006.956309 Case 2. If /f1 is a broken symlink, utimes() returns error and the times are incorrectly set to the current times: err utimes = -1 f1 atime: 2006.956309 f1 mtime: 2006.956309 _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: P_WEXIT
by Terry Lambert 16 Dec '06

16 Dec '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Dec 15, 2006, at 7:04 PM, Joseph Oreste Bruni wrote: On Dec 15, 2006, at 7:35 PM, Terry Lambert wrote: <http://developer.apple.com/technotes/tn2004/tn2118.html> 1) create N number of threads at startup 2) do network and file I/O for days, weeks, etc. 3) receive SIGTERM 4) join with N number of threads 5) exit() 6) hang in kernel who thinks there are still more threads 7) ??? -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On Dec 15, 2006, at 9:20 AM, Quinn wrote: At 20:24 -0700 13/12/06, Joseph Oreste Bruni wrote: On my test system, I can start and kill my process all day long and never reproduce this. In production, where the system runs for weeks, the condition occurs... You might want to set up kernel core dumps on your production machine. When you get a stuck process, you can press NMI to dump a kernel core. From that you could look at the state of the stuck process. I concur; I meant to respond to this earlier, and my opinion is that there's nothing for this except the kernel debugger. If you have a second machine to debug with, then that wold be better thn a core dump server (unless there was an intent to send the core to Apple in a bug report), since a lot of the kgmacros that would be useful for debugging this issue do not work on core files because of gdb limitations that have not been addresses (specifically local shadow register state being used in place of image register state so things like "switchtoact" actually switch). -- Terry I had already opened a bug (4680335), but it was closed with a "behaves correctly" status. Create a new one or can I file this under the original? I figured this would eventually end up being a kernel problem since the basic outline of my program is: I've had to send kernel dumps before so I'm familiar with the process. (Bug in original PCI fibrechannel card in G5 Xserve.) Now I get to figure out how to generate an NMI on the headless Xserve... The guy in charge of the developer interaction shut it down after 7 days of not hearing anything to a ping. The statement in it is "Will re-open if Developer writes back in.". If you set the debug flags for kernel debugging (e.g. 0x144), momentarily hitting the power button will generate the require NMI to drop you into the debugger (or send the core to a dump server, if you have a dump server configured, instead, in the flags), rather than putting the machine to sleep. Let me say up front that I don't believe this is a kernel problem, and the "showallstacks" command should confirm that there are still stacks running in the process. You may end up (if it has already been taken off the active process list) having to track down the task from the zombie list, if it doesn't show up in the task list - meaning it won't show in the "showallstacks". If that's the case, then walk the list to find the process, then look in that for the task, and then "showtaskstacks 0xdsfgsdfg", where the argument is the hex address of the task. Also, if that's the case, it's probably easier to NMI before you shut down the process, get the information from "showalltasks", "c" (continue) it, cause the problem, NMI it again, and then use the information you got previously for the "showtaskstacks" There will be at least one thread in a strange state, and whatever it (or they, if there's more than one) blocks on is the source of your problem. It's be a lot easier if I could attach the machine (8-)), but I really don't have the hardware or software resources necessary to recreate your problem and diagnose it in house. This email sent to site_archiver(a)lists.apple.com

1 0

Re: P_WEXIT
by Joseph Oreste Bruni 16 Dec '06

16 Dec '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Dec 15, 2006, at 7:35 PM, Terry Lambert wrote: <http://developer.apple.com/technotes/tn2004/tn2118.html> 1) create N number of threads at startup 2) do network and file I/O for days, weeks, etc. 3) receive SIGTERM 4) join with N number of threads 5) exit() 6) hang in kernel who thinks there are still more threads 7) ??? Now I get to figure out how to generate an NMI on the headless Xserve... _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com On Dec 15, 2006, at 9:20 AM, Quinn wrote: At 20:24 -0700 13/12/06, Joseph Oreste Bruni wrote: On my test system, I can start and kill my process all day long and never reproduce this. In production, where the system runs for weeks, the condition occurs... You might want to set up kernel core dumps on your production machine. When you get a stuck process, you can press NMI to dump a kernel core. From that you could look at the state of the stuck process. I concur; I meant to respond to this earlier, and my opinion is that there's nothing for this except the kernel debugger. If you have a second machine to debug with, then that wold be better thn a core dump server (unless there was an intent to send the core to Apple in a bug report), since a lot of the kgmacros that would be useful for debugging this issue do not work on core files because of gdb limitations that have not been addresses (specifically local shadow register state being used in place of image register state so things like "switchtoact" actually switch). -- Terry I had already opened a bug (4680335), but it was closed with a "behaves correctly" status. Create a new one or can I file this under the original? I figured this would eventually end up being a kernel problem since the basic outline of my program is: I've had to send kernel dumps before so I'm familiar with the process. (Bug in original PCI fibrechannel card in G5 Xserve.) smime.p7s

1 0

Re: P_WEXIT
by Terry Lambert 16 Dec '06

16 Dec '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com <http://developer.apple.com/technotes/tn2004/tn2118.html> -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On Dec 15, 2006, at 9:20 AM, Quinn wrote: At 20:24 -0700 13/12/06, Joseph Oreste Bruni wrote: On my test system, I can start and kill my process all day long and never reproduce this. In production, where the system runs for weeks, the condition occurs... You might want to set up kernel core dumps on your production machine. When you get a stuck process, you can press NMI to dump a kernel core. From that you could look at the state of the stuck process. I concur; I meant to respond to this earlier, and my opinion is that there's nothing for this except the kernel debugger. If you have a second machine to debug with, then that wold be better thn a core dump server (unless there was an intent to send the core to Apple in a bug report), since a lot of the kgmacros that would be useful for debugging this issue do not work on core files because of gdb limitations that have not been addresses (specifically local shadow register state being used in place of image register state so things like "switchtoact" actually switch). This email sent to site_archiver(a)lists.apple.com

1 0

Re: kgmacros, showallmtx
by Derek Kumar 15 Dec '06

15 Dec '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Derek On Dec 15, 2006, at 12:07 PM, Matt Darland wrote: ---- Thanks, Matt _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/drk%40apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Correct, those fields indicate the number of acquisition attempts, acquisition attempt misses, and the number of misses which caused the acquisition attempt to block respectively (essentially, the mutex routines can spin briefly before blocking if the thread holding the mutex is active on another processor). However, these lock contention statistics are gathered only if either the kernel was booted with the "lcks=3" boot-arg (this enables statistics gathering for all lock groups), or if the statistics and debug attributes have been enabled programmatically for a particular lock group (via lck_grp_attr_setstat) and the mutex (via lck_attr_setdebug). Note that this applies to lck_mtx_lock and variants; the deprecated mutex_ style routines don't collect contention statistics. Also, and whilst I've addressed this for Leopard/x86 (along with additional statistics), the asm mutex implementations on the 10.4 series x86 kernels don't gather contention statistics (but the PowerPC implementations do). There is an interface to collect these statistics from user space, and a program to do so (lockstat, which has options such as periodic stats delta display), but I don't think it's been packaged up for external use. If you think you'll find that useful, feel free to contact me off-list and I'll send you a copy. What do the 'util', 'miss', and 'wait' fields indicate in the showallmtx macro of the kgmacros package? Does this give an idea as to the relative level of contention for locks within a lock group? This email sent to drk(a)apple.com This email sent to site_archiver(a)lists.apple.com

1 0

Re: P_WEXIT
by Quinn 15 Dec '06

15 Dec '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com <http://developer.apple.com/technotes/tn2004/tn2118.html> S+E -- Quinn "The Eskimo!" <http://www.apple.com/developer/> Apple Developer Relations, Developer Technical Support, Core OS/Hardware _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… At 20:24 -0700 13/12/06, Joseph Oreste Bruni wrote: On my test system, I can start and kill my process all day long and never reproduce this. In production, where the system runs for weeks, the condition occurs... You might want to set up kernel core dumps on your production machine. When you get a stuck process, you can press NMI to dump a kernel core. From that you could look at the state of the stuck process. This email sent to site_archiver(a)lists.apple.com

1 0

kgmacros, showallmtx
by Matt Darland 15 Dec '06

15 Dec '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com ---- Thanks, Matt _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… What do the 'util', 'miss', and 'wait' fields indicate in the showallmtx macro of the kgmacros package? Does this give an idea as to the relative level of contention for locks within a lock group? This email sent to site_archiver(a)lists.apple.com

1 0

P_WEXIT
by Joseph Oreste Bruni 14 Dec '06

14 Dec '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com In a previous email, written 15 August 2006 by Terry Lambert: Frustrated, Joseph PS: Best Wishes for the Holidays everyone! _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com The P_WEXIT is set when a process has explicitly called exit(), or has had exit1() called on it, e.g. as a result of taking a fatal signal (either a SIGKILL, or another signal whose default behaviour is to terminate the process), or as a result of proc_shutdown(), which is called on a reboot. It can also be called on a process that has protected itself from being traced, if you attempt to attach a trace to it after it has made itself immune from tracing. The process remains in this state until all active threads have drained out of the process, and the thread that called the exit() (or just the last thread, if exit1() was called on behalf of someone else, or the process was signalled) drains out to the user/ kernel boundary, at which point, if the parent process is not ignoring SIGCHLD, then a zombie structure is allocated, and the contents filled un, and it hangs around until the parent process reaps it by calling one of the wait() functions (e.g. wait4()). If the parent process is ignoring SIGCHLD, then the process does not create a zombie, and is immediately reaped. I've been reading and re-reading this to try to understand why my process is getting stuck. As I might have pointed out in my previous email, by the time my main thread calls exit() (implicitly by returning from main()), there are NO other threads besides the main thread since the main thread has joined with all other threads. Is it possible that the kernel thinks there are still some threads remaining? Could it be related to the fact that my threads are exiting via pthread_testcancel() rather than by pthread_exit()? I know that thread cancelation on Darwin is a bit broken by default without specifying POSIX code via the DARWIN_ALIAS macro. Could I have stumbled onto some part of the kernel that just hasn't received a lot of testing? On my test system, I can start and kill my process all day long and never reproduce this. In production, where the system runs for weeks, the condition occurs... smime.p7s

1 0

P_WEXIT
by Joseph Oreste Bruni 13 Dec '06

13 Dec '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Hello list, Joe Bruni _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com Back in October we had a short discussion involving processes getting stuck with a P_WEXIT status flag. My process is multi-threaded and long-running (as in weeks) and handles up to 2000 simultaneous SSL connections. Our discussion focused on the potential problem with a thread blocking on a call to accept() and how it is not necessarily woken if a second thread close()'s the fd out from under the sleeping thread, especially if there are pending connections. The remedy for this recommended by Terry Lambert was to use pthread_kill() to signal the sleeping thread to wake which can then explicitly use pthread_testcancel() as part of its loop. (This helped to remove a non-portable implementation in the code. Thanks, Terry.) I have re-implemented all connection handling threads in this manner using select() as my blocking point with a timeout of two seconds. I also use SIGUSR1 from the main thread to wake threads blocking (probably overkill since they will wake in two seconds anyway). The program, at shutdown when receiving a SIGTERM, properly pthread_joins() with all threads. All sockets have been closed. I have been painstakingly logging each resource's entire life-cycle from startup to shutdown and I have not missed anything. No memory leaks, no thread leaks, no socket leaks. After returning from main(), as noted from a parting call to syslog() prior to my "return" statement, the process still hangs with a P_WEXIT status flag, occasionally. I'm convinced this is a kernel bug since I'm hanging just outside of main(). My process is started and monitored using launchd, but manually started processes hang this way, too. If it's relevant, this is a mostly-POSIX C++ daemon which links with the OpenSSL library. The OS is 10.4.7 ppc on a dual G5 (we are scheduling to update to 10.4.8 soon). One other thing I'd like to bring up that I forgot initially: Awhile back I had a bug in the usage of an STL list that wasn't properly mutexed based on my mis-interpretation of STL and thread-safety. This was occasionally causing my program to segfault. Oddly, even a segfault would land my program in the P_WEXIT state!! (I have since mutex'd access to the list.) Are there any other suggestions? This one is very difficult to troubleshoot since the hang is outside my code. I'm willing to do a ktrace during the shutdown if some one thinks it will help. smime.p7s

1 0