April 2007 - Darwin-dev - lists.apple.com

Re: BSD sockets ::select() performance on 10.4.9
by Steve Checkoway 10 Apr '07

10 Apr '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Apr 10, 2007, at 3:34 PM, Terry Simons wrote: On Apr 10, 2007, at 3:27 PM, Steve Checkoway wrote: -- Steve Checkoway _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Apr 10, 2007, at 11:22 AM, Terry Simons wrote: You could impose some sort of artificial sleep condition every now and then to force your application to yield the CPU, but that could cause performance issues too. As long as there is no data to read and you are not writing, select should block without needing any sleep. But if your application is getting bombarded with data, select will return often and could possibly cause the CPU to grind, right? I suppose it could but chances are you are wanting to do something with that data which is probably more expensive than a system call. This email sent to site_archiver(a)lists.apple.com

1 0

Re: BSD sockets ::select() performance on 10.4.9
by Terry Simons 10 Apr '07

10 Apr '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Apr 10, 2007, at 3:27 PM, Steve Checkoway wrote: On Apr 10, 2007, at 11:22 AM, Terry Simons wrote: Right. ;) - Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Are you using exception/write fd sets, or just the read sets? In one of my applications I initially had write sets turned on, and saw a CPU spike, presumably because select was returning immediately due to at least one of the descriptors being writable (which is almost always the case). When select returns, you need to read data from each descriptor in the read set and write data to each descriptor in the write set otherwise select will return immediately. In my case I was never writing to the descriptors... so it was returning immediately. My application doesn't have a lot of data, though, so removing the write sets was enough to settle things down. If you aren't planning on writing, the descriptors should not be added to the set. You could impose some sort of artificial sleep condition every now and then to force your application to yield the CPU, but that could cause performance issues too. As long as there is no data to read and you are not writing, select should block without needing any sleep. But if your application is getting bombarded with data, select will return often and could possibly cause the CPU to grind, right? This email sent to site_archiver(a)lists.apple.com

1 0

Re: ::select() and threads
by Josh Graessley 10 Apr '07

10 Apr '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com -josh On Apr 10, 2007, at 2:05 PM, jmzorko(a)mac.com wrote: Hello all ... ml_set_interrupts_enabled thread_block_reason thread_block uiomove selprocess select Regards, John Falling You - exploring the beauty of voice and sound http://www.fallingyou.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com This just means that when shark sampled your process, that thread was blocked (not running) waiting for an event. I've a question -- under Darwin, is select() used to determine which threads are ready to run? Shark is reporting that our app spends most of its' time in select(), but the callstack that represents the majority of select() calls isn't our select() loop, it's something else: ... which makes me think that the kernel is using select() to determine when threads are ready. This makes some sense, as our app is very multithreaded. If this is the case, how can I get our (possibly too) multithreaded app to be kinder to the CPU under OSX? _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/jgraessley% 40apple.com This email sent to jgraessley(a)apple.com smime.p7s

1 0

Re: BSD sockets ::select() performance on 10.4.9
by Steve Checkoway 10 Apr '07

10 Apr '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Apr 10, 2007, at 11:22 AM, Terry Simons wrote: -- Steve Checkoway _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Are you using exception/write fd sets, or just the read sets? In one of my applications I initially had write sets turned on, and saw a CPU spike, presumably because select was returning immediately due to at least one of the descriptors being writable (which is almost always the case). When select returns, you need to read data from each descriptor in the read set and write data to each descriptor in the write set otherwise select will return immediately. My application doesn't have a lot of data, though, so removing the write sets was enough to settle things down. If you aren't planning on writing, the descriptors should not be added to the set. You could impose some sort of artificial sleep condition every now and then to force your application to yield the CPU, but that could cause performance issues too. As long as there is no data to read and you are not writing, select should block without needing any sleep. This email sent to site_archiver(a)lists.apple.com

1 0

::select() and threads
by jmzorko＠mac.com 10 Apr '07

10 Apr '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Hello all ... ml_set_interrupts_enabled thread_block_reason thread_block uiomove selprocess select Regards, John Falling You - exploring the beauty of voice and sound http://www.fallingyou.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I've a question -- under Darwin, is select() used to determine which threads are ready to run? Shark is reporting that our app spends most of its' time in select(), but the callstack that represents the majority of select() calls isn't our select() loop, it's something else: ... which makes me think that the kernel is using select() to determine when threads are ready. This makes some sense, as our app is very multithreaded. If this is the case, how can I get our (possibly too) multithreaded app to be kinder to the CPU under OSX? This email sent to site_archiver(a)lists.apple.com

1 0

Re: BSD sockets ::select() performance on 10.4.9
by Terry Simons 10 Apr '07

10 Apr '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Hi John, - Terry On Apr 10, 2007, at 11:14 AM, jmzorko(a)mac.com wrote: Hello, all ... My questions: 1. Am I interpreting the Shark output correctly? 2. If so, how can I make select() perform better in this case? Regards, John Falling You - exploring the beauty of voice and sound http://www.fallingyou.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/galimore%40mac.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… If you always have data to read, select will return immediately and cause your application to process the data... I don't think there is a way to make select behave differently in this case. Are you using exception/write fd sets, or just the read sets? In one of my applications I initially had write sets turned on, and saw a CPU spike, presumably because select was returning immediately due to at least one of the descriptors being writable (which is almost always the case). My application doesn't have a lot of data, though, so removing the write sets was enough to settle things down. You could impose some sort of artificial sleep condition every now and then to force your application to yield the CPU, but that could cause performance issues too. I'm definitely no expert, though, so maybe someone else has better input. I'm working on a network application that runs on Windows as well as Mac OSX. It's written in C++, and 99% of the code is portable between the two operating systems. However, the OSX version currently uses a lot more (read: 4-5x or more) CPU than the Windows version, and i'm trying to figure out why. So, I start the app and then run Shark, selecting "Time Profile". What it seems to be telling me is that this app spends most of its' time (50%) in ::select(). This makes sense, as this app communicates over the network a lot to other nodes, though this test is run locally e.g. it's not trying to connect to anyone else and is just waiting for someone to connect to it. Still, it makes me wonder how I can optimize it, while remaining cross-platform (platform-specific code is allowed, though we try to keep it to a minimum, as we may have to port this to Linux and possibly the PS3 soon). However, the weird thing is that, when ::select() is using most of the CPU, it's called from selprocess() / uiomove() / thread_block () / thread_block_reason() / ml_set_interrupts_enabled(). I've no idea what that stuff is, so I guess I should go looking in the kernel. Is this the same ::select() that I use when looking for ready sockets? Shark seems to be telling me that it is (it points to the same address as ::select() when called from my select loop). If so, is there a way I can optimize its' use when called via selprocess() / uiomove() / thread_block() / thread_block_reason () / ml_set_interrupts_enabled() so it doesn't use so much CPU? This email sent to galimore(a)mac.com This email sent to site_archiver(a)lists.apple.com

1 0

BSD sockets ::select() performance on 10.4.9
by jmzorko＠mac.com 10 Apr '07

10 Apr '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Hello, all ... My questions: 1. Am I interpreting the Shark output correctly? 2. If so, how can I make select() perform better in this case? Regards, John Falling You - exploring the beauty of voice and sound http://www.fallingyou.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I'm working on a network application that runs on Windows as well as Mac OSX. It's written in C++, and 99% of the code is portable between the two operating systems. However, the OSX version currently uses a lot more (read: 4-5x or more) CPU than the Windows version, and i'm trying to figure out why. So, I start the app and then run Shark, selecting "Time Profile". What it seems to be telling me is that this app spends most of its' time (50%) in ::select(). This makes sense, as this app communicates over the network a lot to other nodes, though this test is run locally e.g. it's not trying to connect to anyone else and is just waiting for someone to connect to it. Still, it makes me wonder how I can optimize it, while remaining cross-platform (platform-specific code is allowed, though we try to keep it to a minimum, as we may have to port this to Linux and possibly the PS3 soon). However, the weird thing is that, when ::select() is using most of the CPU, it's called from selprocess() / uiomove() / thread_block() / thread_block_reason() / ml_set_interrupts_enabled(). I've no idea what that stuff is, so I guess I should go looking in the kernel. Is this the same ::select() that I use when looking for ready sockets? Shark seems to be telling me that it is (it points to the same address as ::select() when called from my select loop). If so, is there a way I can optimize its' use when called via selprocess() / uiomove() / thread_block() / thread_block_reason() / ml_set_interrupts_enabled() so it doesn't use so much CPU? This email sent to site_archiver(a)lists.apple.com

1 0

Re: Detecting case-sensitivity of a filesystem
by Brian Bergstrand 10 Apr '07

10 Apr '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Apr 9, 2007, at 11:51 PM, Nathaniel Gray wrote: On Apr 9, 2007, at 4:57 PM, Nathaniel Gray wrote: struct attrlist alist; bzero(&alist, sizeof(alist)); alist.volattr = ATTR_VOL_CAPABILITIES; unsigned char buffer[sizeof(vol_capabilities_attr_t) + sizeof(size_t)]; ... // See man page for how to setup and interpret buffer err = getattrlist(path, &alist, buffer, sizeof(buffer), 0); if (!err && (alist.volattr & ATTR_VOL_CAPABILITIES) { vol_capabilities_attr_t *vcaps; // cast buffer to caps if (vcaps->capabilities[0] & VOL_CAP_FMT_CASE_SENSITIVE) { ; // vol is case sensitive } else { ; // vol is case insensitive } } Brian Bergstrand <http://www.bergstrand.org/brian/> PGP Key ID: 0xB6C7B6A2 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iD8DBQFGG7idedHYW7bHtqIRAr3xAKDTIok3XB/ihtUeOC+TZaEeaijAlgCgrcFc /60OlgVEUV1NLlssPnL3+/0= =TMR7 -----END PGP SIGNATURE----- _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Apr 9, 2007, at 6:04 PM, Don Brady wrote: Is there any non-hackish way to detect the case-sensitivity of a filesystem, like a system call? statvfs doesn't appear to do it and statfs doesn't help either (case-sensitive and case- insensitive hfs+ have the same f_type and f_fstypename). Or is there no avoiding the "create foo, open Foo" hack? You can use pathconf(2) with the _PC_CASE_SENSITIVE selector. Note an EINVAL result for this selector generally implies case- sensitive. This works nicely for hfs+ in the case-sensitive and insensitive variants. Unfortunately it looks like EINVAL is returned by smbfs and fat32, which are both case-insensitive, and ufs, which is case- sensitive. So I guess this test is only reliable when EINVAL is not returned (probably just for hfs-family filesystems). Any other suggestions? You can get this info via the getattrlist() syscall (std disclaimer concerning code written in Mail applies): This email sent to site_archiver(a)lists.apple.com

1 0

Re: Detecting case-sensitivity of a filesystem
by Pelle Johansson 10 Apr '07

10 Apr '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com (sorry for double-send, wrong email on previous) 10 apr 2007 kl. 00.57 skrev Nathaniel Gray: _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Is there any non-hackish way to detect the case-sensitivity of a filesystem, like a system call? statvfs doesn't appear to do it and statfs doesn't help either (case-sensitive and case-insensitive hfs+ have the same f_type and f_fstypename). Or is there no avoiding the "create foo, open Foo" hack? diskutil seems to get this information from statfs, according to ktrace (the only relevant syscalls I could find). I'm not sure if it's part of Darwin, but in that case maybe you can simply parse it's output? The text it outputs comes from /System/Library/Filesystems/ hfs.fs/Contents/Resources/English.lproj/InfoPlist.strings or (less likely) /System/Library/Filesystems/hfs.fs/Contents/Info.plist, so you might have to watch out for localisation. -- Pelle Johansson This email sent to site_archiver(a)lists.apple.com

1 0

Re: Detecting case-sensitivity of a filesystem
by Andrew White 10 Apr '07

10 Apr '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com User-agent: Thunderbird 1.5.0.10 (Macintosh/20070221) -- Andrew White _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Nathaniel Gray wrote: On Apr 9, 2007, at 6:04 PM, Don Brady wrote: Any other suggestions? Why do you want to know / what are you trying to achieve? If we know the actual goal there might be a workaround. This email sent to site_archiver(a)lists.apple.com

1 0