August 2009 - Darwin-dev - lists.apple.com

Re: CFFileDescriptor fd leak?
by A.M. 20 Aug '09

20 Aug '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Aug 20, 2009, at 6:47 PM, Dave Keck wrote: Hey, #include <CoreFoundation/CoreFoundation.h> #include <unistd.h> #include <sys/event.h> static void junkCallback(CFFileDescriptorRef fdref, CFOptionFlags callBackTypes, void *info) {} int main(int argc, char *argv[]) { int i = 0; for (i = 0; i < 3; i++) { CFFileDescriptorRef fileDescriptor = nil; CFRunLoopSourceRef runLoopSource = nil; NSLog(@"CREATE %d", i); fileDescriptor = CFFileDescriptorCreate(NULL, 0, false, junkCallback, NULL); runLoopSource = CFFileDescriptorCreateRunLoopSource(NULL, fileDescriptor, 0); CFRunLoopAddSource([[NSRunLoop currentRunLoop] getCFRunLoop], runLoopSource, kCFRunLoopDefaultMode); } sleep(-1); return 0; } ============================== Hi Dave, #include <CoreFoundation/CoreFoundation.h> #include <unistd.h> #include <sys/event.h> static void junkCallback(CFFileDescriptorRef fdref, CFOptionFlags callBackTypes, void *info) {} struct __FileDescriptorHackStruct { int junkA; int junkB; int junkC; int internalKqueueDescriptor; }; int main(int argc, char *argv[]) { int i = 0; for (i = 0; i < 3; i++) { CFFileDescriptorRef fileDescriptor = nil; CFRunLoopSourceRef runLoopSource = nil; fileDescriptor = CFFileDescriptorCreate(NULL, 0, false, junkCallback, NULL); runLoopSource = CFFileDescriptorCreateRunLoopSource(NULL, fileDescriptor, 0); CFRunLoopAddSource(CFRunLoopGetCurrent(), runLoopSource, kCFRunLoopDefaultMode); CFRunLoopRemoveSource(CFRunLoopGetCurrent(), runLoopSource, kCFRunLoopDefaultMode); CFRelease(runLoopSource); CFFileDescriptorInvalidate(fileDescriptor); CFRelease(fileDescriptor); } sleep(-1); return 0; } Cheers, M -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.10 (Darwin) iEYEARECAAYFAkqN328ACgkQqVAj6JpR7t4LUQCfamtzP0xV6GWEBNxpQRw6ZK+I SuAAn3wouHd+LXixOKheQFJL3ccy2IuI =iogu -----END PGP SIGNATURE----- _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I ran your test code and saw the same results. After tweaking the code a bit, I found that to have the kqueues cleaned up, you have to create a run loop source for the CFFD, add it to the run loop, and then remove it from the run loop. Here's the tweaked code: ============================== NSLog(@"DESTROY %d", i); CFRunLoopRemoveSource([[NSRunLoop currentRunLoop] getCFRunLoop], runLoopSource, kCFRunLoopDefaultMode); CFRelease(runLoopSource); CFFileDescriptorInvalidate(fileDescriptor); CFRelease(fileDescriptor); I'd definitely consider your findings a bug. But I don't think it poses any real issue, since CFFileDescriptor is useless unless it's used with a run loop. I'd say a radar is in order... I shaved down the code from the previous email from code which did indeed add the CFFileDescriptor to the runloop and leaked. However, I did not remove the source from the runloop manually like you do above and the Apple example code (http://developer.apple.com/documentation/CoreFoundation/Reference/CFFileDes… ) doesn't remove the runloop source either. Unfortunately, the code you laid out above does indeed sometimes leak a single kqueue fd (OS 10.5.8): rd07:~ alex$ ./davekeck & [1] 4880 rd07:~ alex$ lsof -p $! | grep KQUEUE davekeck 4880 alex 4u KQUEUE count=0, state=0 If I run it with breakpoints on kqueue, close, and kevent in gdb, then the leak disappears, which indicates that the Apple code has a race condition. I noticed that close() is called in the __monitor_file_descriptor__ thread, so I suspect that without the thread created from the addition of a source, that close on the kqueue will never be called. The above code leaks sometimes due to a race condition in inter-thread communications- bizarrely, it leaks descriptor 4 which is the first that was allocated- I would have thought that the last one would leak... With the following hack-around code (based on yours), the fd leak is permanently squelched: close(((struct __FileDescriptorHackStruct*)fileDescriptor)- >internalKqueueDescriptor); At least it is not leaking threads. Speaking of which, why does each file descriptor need to be watched by a new thread? Not even CFSocketManager is that dumb. In any case, this is looking so bad that I will likely need to implement my own CFFileDescriptor replacement. This email sent to site_archiver(a)lists.apple.com

1 0

Re: which pages of the file are in the cache
by Eric Gouriou 20 Aug '09

20 Aug '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: which pages of the file are in the cache
by Joel Reymont 20 Aug '09

20 Aug '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:cc:message-id:from:to :in-reply-to:content-type:content-transfer-encoding:mime-version :subject:date:references:x-mailer; bh=T/NVoKfkzAqpdOtp0tagjmJ+foXXLKY07V7sWPOeVxQ=; b=rRnUGm3eAFDpE+rO3htMYJbGvMNHwhJGzYD6vqkJR5AAvG/S2lwKXsv0kroarFNBKm HtYy1TTjeG3sy8G/OPFr/4jBApgkf6+aVsy9HlD5q2MaOkP25h5a7XTLKTMLdAH0PJKl G7oIIwfObxrvadDaOTEsI46afdNJ8hQf5y5MY= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=cc:message-id:from:to:in-reply-to:content-type :content-transfer-encoding:mime-version:subject:date:references :x-mailer; b=inBnUlS9wJbjV9ctiUZWX9eJbj+RyaTn1IuO6Ffxif2we2/eZY6AC6uZepqGDyujUR YwBLX3cKcWYgDQJrZ5R0T4w1a2aIz5UDIddwpPQ57xKagODBpAOs3b82N46UKMreGw9P uU+4lgtasof+7EN0wyZRvBL/FWafgDHQSoGhI= On Aug 20, 2009, at 5:46 PM, Eric Gouriou wrote: grep "dyld: loaded" xx|awk '{print $3}'|xargs otool -L > yyy grep OpenGL yyy Thanks, Joel [1] http://pastie.org/590506 --- faster mac firefox http://tinyco.de _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Remove load-time initializers (reduce their number, make them compile-time constants or move them into static initializers in accessor functions to get them lazily executed) According to my latest test run [1], I have 182 XUL initializers and 101 OpenGL ones. I'm at a loss as to who is pulling in OpenGL, e.g. the following does not reveal it Is there a way to move OpenGL initialization into the future and out of the startup? Intuitively, I don't think it can be done because I think all Mac drawing is OpenGL-based. Still, I thought I'd confirm. This email sent to site_archiver(a)lists.apple.com

1 0

Re: CFFileDescriptor fd leak?
by A.M. 20 Aug '09

20 Aug '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Aug 20, 2009, at 1:58 PM, A.M. wrote: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello, #include <CoreFoundation/CoreFoundation.h> #include <unistd.h> #include <sys/event.h> struct __FileDescriptorHackStruct { int junkA; int junkB; int junkC; int internalKqueueDescriptor; }; Cheers, M -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.10 (Darwin) iEYEARECAAYFAkqN0pQACgkQqVAj6JpR7t5WkACgqpXpWadi9b492b2VsrmHmaaT R8MAoLC49eH9DdYLWXygxOavnVZO218K =OBAa -----END PGP SIGNATURE----- _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I am using CFFileDescriptor and it noticed that it seems to leak its kqueue file descriptor. Here is sample code demonstrating the problem: Wow. So I managed to cook up an evil hack which properly closes the kqueue: static void junkCallback(CFFileDescriptorRef fdref, CFOptionFlags callBackTypes, void *info) {} int main(int argc, char *argv[]) { int i; for(i=0;i<3;i++) { CFFileDescriptorRef fdref = CFFileDescriptorCreate(kCFAllocatorDefault, 0, false, junkCallback, NULL); printf("%d\n",((struct __FileDescriptorHackStruct*)fdref)- >internalKqueueDescriptor); close(((struct __FileDescriptorHackStruct*)fdref)- >internalKqueueDescriptor); CFFileDescriptorInvalidate(fdref); CFRelease(fdref); } sleep(100000); return 0; } If one flips the CFFileDescriptorInvalidate() and close() lines, the kqueue fd stays open. At least I can stop the leakage now- any ideas? This email sent to site_archiver(a)lists.apple.com

1 0

Re: CFFileDescriptor fd leak?
by Dave Keck 20 Aug '09

20 Aug '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:from:date:x-google-sender-auth:message-id:subject:to:cc :content-type; bh=HsyZcVoZCbpon0yPnADgI4KQ41JrZos+kLRFpn7UX+0=; b=U1wmDzG0LduuPam8tWCH2pIRqN6VVc36s16LHdhzmDPV0ag4WFmXI7KO6M65S//rKN Qo/zyMpQm8LLpRuIN+faYf4TqTq9kqDfb6Xqb3f5CaLkUH3NzA2c6pQEwfNbrLqh3P58 sHqN/0r2Z0j8Y0fUAXxhlxHyohLg5pqvE7s7Y= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type; b=lY7oDvyDWgvb+pSvUiXoRI/3mirVoWRqYDy8nfLB++fs1CZ0SNmUNbepRJr7ZxqL3s ArcfS2ajQUwwxtYRtSiE2pvQI/fdRfkukUBkd5NssHzxz+TwsaI33b98f5QCBp7KncDC 4LHf2OMkBRgos+NsCqjUKLvWhxtvclHA6Li3w= _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: which pages of the file are in the cache
by Joel Reymont 20 Aug '09

20 Aug '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:cc:message-id:from:to :in-reply-to:content-type:content-transfer-encoding:mime-version :subject:date:references:x-mailer; bh=JorfDBVyea4A142SoBqmft6ScBlQOJjIbSQAgW+9lIA=; b=vw5+LxGAWag97FX+Z4WxUgf5k0o+W+n9nEGwp1W9xjEFFCBIK/JPXaOLo/r1bZFPw7 lN5v6+WTxe4yAOcrUZF60PlBEzdFnrviBbRwaFmuzsTTleiBdgOrnTveVAWoiKP89Xxv IvBtxXQPlQYwDhvkEC7oA4LgZ5SEO/htVI5W0= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=cc:message-id:from:to:in-reply-to:content-type :content-transfer-encoding:mime-version:subject:date:references :x-mailer; b=M+GnC2CJTLkkXlBS3u+BuDmlivgnexrOffaky52pIM42/v3gokOZApSnwlxtU+AtBZ 8NsS3BgJPqyXKXLxexymgmdppH4mlqZf0htPPD1i8TxkOSiTv98Q9ruJyB9Dv5AWACOK +H3RCQLczwm+Xoxrgie4xVgoVfdvTZNd+2AQo= On Aug 20, 2009, at 5:46 PM, Eric Gouriou wrote: Do initializers go into total init time? Thanks, Joel [1] http://gist.github.com/171379 --- faster mac firefox http://tinyco.de _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… To get a better idea of how long the dynamic linking takes, I recommend using DYLD_PRINT_STATISTICS I ran with extra dyld options to print libraries loaded as well as initializers and got plenty of information [1]. The statistics are as follows: total time: 1.3 seconds (100.0%) total images loaded: 106 (93 from dyld shared cache, 56 needed no fixups) total segments mapped: 49, into 5894 pages with 684 pages pre-fetched total images loading time: 328.8 milliseconds (24.1%) total rebase fixups: 148,829 total rebase fixups time: 4.0 milliseconds (0.2%) total binding fixups: 24,863 total binding symbol lookups: 796, average images searched per symbol: 2.3 total binding fixups time: 112.2 milliseconds (8.2%) total bindings lazily fixed up: 45 of 19,103 total init time time: 919.3 milliseconds (67.3%) total images with weak exports: 3 It seems that the binding fixup time can be discounted (8.2%). I think the image loading time (24.1%) and total initialization time (67.3%) can be improved upon, though. This email sent to site_archiver(a)lists.apple.com

1 0

Re: List of accounts
by Axel Luttgens 20 Aug '09

20 Aug '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Le 20 août 09 à 16:00, Tim Murison a écrit : If you want just local accounts that can log in, use dscl /Local/Default -search Users PrimaryGroupID 20 Thanks for the suggestion. No. That would be too easy... ;-) I guess the crux of my question would be, is this what Apple uses to populate the accounts system preferences dialog? The real question is: what do you mean by "being able to log in"? HTH, Axel _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Does the system require that a user be a member of the gid=20 ("Users") group to log in? [...] For example, you asked about the AuthenticationAuthority attribute. Even without such an attribute in his record, a user might be able to authenticate: when absent, the AuthenticationAuthority attribute defaults to "Basic". A basic authentication scheme means that the user's password is expected to be stored as a traditional unix salted- encrypted into the Password attribute of the user's record. On the other hand, a user's account may be disabled with the appearance of a "Disabled" value in the AuthenticationAuthority attribute; but an account may also be disabled through the Password Server, should that account be governed by the Password Server. Now come the conventions traditionally adopted by the various services. For example, for being able to open a ftp session, the user's shell must appear in /etc/shells. On the other hand, an ssh session could be opened with the help of a public key. Then come the restrictions implemented by Apple's conventions. As Geoff already wrote, depending on the server's settings, a user may be required to belong to certain groups (com.apple.access_ssh, com.apple.acess_loginwindow...) so as to be able to access certain services. Finally, there are those services that just don't go through the system and create their own users and manage authentication according to their own rules. As you already noticed,a MySQL user is ... a MySQL user, and the system doesn't know anything about that user. The same way, Postfix may handle its own users, and they don't need to appear in some kind of system directory. Given all those possibilities, does your original question still hold? As a reminder: "I'm trying to determine how to get a list of BSD user ids that can login to the machine, either with a GUI or via ssh". This email sent to site_archiver(a)lists.apple.com

1 0

CFFileDescriptor fd leak?
by A.M. 20 Aug '09

20 Aug '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello, //gcc test.c -framework CoreFoundation -g #include <CoreFoundation/CoreFoundation.h> #include <unistd.h> #include <sys/event.h> What am I missing? Cheers, M -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.10 (Darwin) iEYEARECAAYFAkqNjt8ACgkQqVAj6JpR7t5BwgCgm2K912i93tviOzkFbhNo8cyJ Cn8AoJ8UOliN1+rsERGhGbbkFPD6uuM7 =xSEK -----END PGP SIGNATURE----- _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I am using CFFileDescriptor and it noticed that it seems to leak its kqueue file descriptor. Here is sample code demonstrating the problem: static void junkCallback(CFFileDescriptorRef fdref, CFOptionFlags callBackTypes, void *info) {} // one argument, an integer pid to watch, required int main(int argc, char *argv[]) { int i; for(i=0;i<3;i++) { CFFileDescriptorRef fdref = CFFileDescriptorCreate(kCFAllocatorDefault, 0, false, junkCallback, NULL); CFFileDescriptorInvalidate(fdref); CFRelease(fdref); //CFRunLoopRunInMode(kCFRunLoopDefaultMode, 1.0, false); // doesn't help to spin the runloop } sleep(100000); return 0; } While the executable is sleeping, I run "lsof -p <pid>" and see the lines for open kqueue descriptors: a.out 725 agentm 3u KQUEUE count=0, state=0 a.out 725 agentm 4u KQUEUE count=0, state=0 a.out 725 agentm 5u KQUEUE count=0, state=0 If I debug the executable, I can plainly see that the kqueue descriptors are opened in CFFileDescriptorCreate() but never closed. Also, the sample code on the documentation page (http://developer.apple.com/documentation/CoreFoundation/Reference/CFFileDes… ) also seems to leak two fds. This email sent to site_archiver(a)lists.apple.com

1 0

Re: which pages of the file are in the cache
by Eric Gouriou 20 Aug '09

20 Aug '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: List of accounts
by Mo McRoberts 20 Aug '09

20 Aug '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Thu, Aug 20, 2009 at 15:00, Tim Murison<tim.murison(a)radialpoint.com> wrote: >> If you want just local accounts that can log in, use >> dscl /Local/Default -search Users PrimaryGroupID 20 > > Thanks for the suggestion. > > Does the system require that a user be a member of the gid=20 ("Users") > group to log in? I guess the crux of my question would be, is this what > Apple uses to populate the accounts system preferences dialog? On my MacBook, with a mobile account: [mo@pip ~]$ dscl /Local/Default -search Users PrimaryGroupID 20 admin PrimaryGroupID = ( 20 ) nonadmin PrimaryGroupID = ( 20 ) sysop PrimaryGroupID = ( 20 ) “admin” is also a mobile account, but has logged into this Mac in the past and possibly has screwed up group membership (my OD master was originally a Linux box running OpenLDAP which was hand-configured). “nonadmin” (a test account) and “sysop” (local administrator account) are both local accounts. My account isn't in the list, however, nor are others in the directory (the default primary group is “everyone”, IIRC). In other words, although it's _possible_ for an AD or OD user to have a primary GID of 20, they wouldn't do ordinarily. Similarly, although it's possible for a local user to have a GID which isn't 20, you have to go to some trouble to configure them that way for no appreciable benefit. M. -- http://nevali.net _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0