July 2007 - Darwin-dev - lists.apple.com

Kernel Symbol Clarification
by Duane Murphy 18 Jul '07

18 Jul '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com I wanted to make sure I understood how symbols interact in kext's. If I use a generally available library and another kext also uses that library, will those symbols interact with each other when the kext are loaded? That is it possible that my kext will accidentally link to the symbols in the other kext using a the same library? Of course the problem is that changes or versions of these libraries are different and could cause problems with either kext. Thank you for this clarification. ...Duane _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Kernel Recursive mutexes 10.4.9 ++
by Erez Kaplan 18 Jul '07

18 Jul '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: need lots of shared memory (with shmget) in a Cocoa app
by Terry Lambert 13 Jul '07

13 Jul '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com -- Terry Here is the line in /etc/rc I changed: I am on this version of Darwin: Hope this was helpful. Norm Green GemStone Systems Inc. http://www.gemstone.com/smalltalk/ -----Original Message----- From: Jay Reynolds Freeman [mailto:jay_reynolds_freeman@mac.com] Sent: Friday, July 13, 2007 1:03 PM To: darwin-dev(a)lists.apple.com list Subject: need lots of shared memory (with shmget) in a Cocoa app This topic overlaps Cocoa and Darwin, bear with me a minute: I would like to make some serious use of interprocess shared memory in Cocoa applications. I don't mean pipelines or their equivalents; my project involves having multiple different applications all making asynchronous reads and writes to different parts of a large (Gigabyte) area of shared memory. (Details later, for the curious.) I would be using shmget, shmat, and so on, as low-level primitives to obtain and access shared memory. I have figured out that OS X doesn't support that much shared memory right out of the box. (BTW,I am running 10.4.10 on a 13-inch Intel Macbook with 1 GByte of physical memory.) I have done enough research to know that some of the parameters that control shared memory are in /etc/rc, which is read at boot-time, and I even managed to find some documentation on the web for what kern.sysv.shm<whatever> means. (The documentation I found was in Spanish, which I don't read or speak, but fortunately the Google translation was usable -- the search experience was sufficiently Monty-Pythonesque that I found myself lifting my feet in case of killer rabbits.) And I found a lot of comments from people who had tried to get shared memory to work on the Mac, and had not been as successful as they had hoped. I hope that is enough homework to warrant an appeal to this group for further pointers. Specifically, 1) Is there any thorough documentation anywhere about how shared memory on the Mac actually works? I need all the background information I can get, and I certainly need to know whether there are hard or soft range limits for the kern.sysv.<whatever> parameters, and what happens if you crowd those limits. Just man pages aren't near enough, and googling and searching XCode documentation hasn't turned up anything interesting yet. 2) What am I likely to break elsewhere in the Mac architecture if I start increasing shared-memory limits? 3) How do the layers of software architecture upon which a Cocoa application runs interact with the shared memory stuff? I haven't tried messing with /etc/rc yet, but in my application, I could not successfully create even a 1 MByte shared-memory segment. I suspect that something else is using enough of the default shared-memory resources that there weren't enough left for my test, and I need to know how much to allow for any such uses. (If this question should go to another group, tell me and I will take it there.) If you are still reading and are curious, what I have is a functioning Lisp system (Wraith Scheme -- see my web site if you are terminally curious (URL below)); I think I have a way to add an access-rights system to it such that multiple copies of the Lisp interpreter can interact with the same heap without stepping on each others' toes. The heap is the big shared memory object I spoke of. (Lisps eat memory as if there were no tomorrow, 1 GByte is not a lot...) Using just separate threads, one for each Lisp interpreter, isn't appropriate, because each interpreter has substantial amounts of truly private data. Just fork (without exec) won't work on a Mac app, at least, not if you want both parent and child to have a functioning GUI. Small numbers of cooperating parallel Lisp processes would be fun, and would be a natural fit to present and likely near-future Mac processor hardware. Enough blathering. Thanks for any hints or suggestions you may have. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/norm.green%40gemstone.com This email sent to norm.green(a)gemstone.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/tlambert%40apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… You are significantly better off using memory sharing via mmap or via Mach primitives than you would be using System V shared memory. The reasoning behind the default imitations have to do with it taking wired memory, which doesn't really happen for mmap or by passing around address spaces between processes using Mach primitives. This also has a number of other benefits, depending on the approach you choose to use: (1) If you use mmap, the data is backed by a file, and will persist over crashes or reboots, which would make it easier to pick up where you left off, if your application supports it; this approach would be ideal for, for example, a ray tracing or other restartable computation model (would require an intention log to know where you are at in the ob). (2) If you use Mach primitives, the size limitation can be overcome by passing region descriptors between 32 and 64 bit applications (wffectively, your 32 bit applications could have a 512M-1G window on a much larger virtual region maintaind by a 64 bit app, as one example). (3) Resource tracking cleanup in either of these approaches (i.e. what you have to do to recover from a crash) is much easier than in System V or POSIX shared memory, since the objects persist past the lifetime of your programs; with SysV or POSIX, you either have to know the id (e.g. use ftok()) and release the memory every time you start up, ignoring the error if it isn't there already). (4) You don't eat wired memory or kernel virtual address space; for a 1G shared memory object, this might actually mean using SysV/POSIX ends up being impossible (in order to persist as a rendezvous, the SysV and POSIX objects pretty much have to end up allocated in a persistent address space that can be mapped out into user processes; hence this limitation). The downside of mmap is that pages will get written to the backing store file under memory pressure; if you don't need the data to be persistent across reboot/program crash, then this ends up costing disk I/O;. On the other hand, given the size of the regions being contemplated here, it's either push it to a backing file or push it to swap: either way you're going to be hitting the disk, unless you have a huge amount of physical RAM, so this might not be an issue for you (enough RAM = no memory pressure = no disk I/O for dirty pages). Note that the above is pretty much true for most OSs (but most OSs don't support Mach primitives for sharing). My personal recommendation is to use mmap; if you want to use the Mach approach, then you will likely want to buy Amit Sing's book to get the documentation necessary for you to be able to write the code. On Jul 13, 2007, at 1:13 PM, Norm Green wrote: Jay, I recently ported our Smalltalk virtual machine and application server to Darwin. Like LISP, Smalltalk uses substantial amounts of heap and shared memory. I had no problems getting 10 MB of shared memory after I increased the default limits in /etc/rc and rebooted. sysctl -w kern.sysv.shmmax=104857600 kern.sysv.shmmin=1 kern.sysv.shmmni=32 kern.sysv.shmseg=8 kern.sysv.shmall=104857600 Darwin Kernel Version 8.9.3: Fri Apr 27 14:50:07 PDT 2007; root:xnu-792.19.5~2/RELEASE_I386 I didn't locate any docs on the subject, I just made the changes above and it worked. -- Jay Reynolds Freeman --------------------- Jay_Reynolds_Freeman(a)mac.com http://web.mac.com/jay_reynolds_freeman (personal web site) This email sent to tlambert(a)apple.com This email sent to site_archiver(a)lists.apple.com

1 0

RE: need lots of shared memory (with shmget) in a Cocoa app
by Norm Green 13 Jul '07

13 Jul '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

need lots of shared memory (with shmget) in a Cocoa app
by Jay Reynolds Freeman 13 Jul '07

13 Jul '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com This topic overlaps Cocoa and Darwin, bear with me a minute: I would like to make some serious use of interprocess shared memory in Cocoa applications. I don't mean pipelines or their equivalents; my project involves having multiple different applications all making asynchronous reads and writes to different parts of a large (Gigabyte) area of shared memory. (Details later, for the curious.) I would be using shmget, shmat, and so on, as low-level primitives to obtain and access shared memory. I have figured out that OS X doesn't support that much shared memory right out of the box. (BTW,I am running 10.4.10 on a 13-inch Intel Macbook with 1 GByte of physical memory.) I have done enough research to know that some of the parameters that control shared memory are in /etc/rc, which is read at boot-time, and I even managed to find some documentation on the web for what kern.sysv.shm<whatever> means. (The documentation I found was in Spanish, which I don't read or speak, but fortunately the Google translation was usable -- the search experience was sufficiently Monty-Pythonesque that I found myself lifting my feet in case of killer rabbits.) And I found a lot of comments from people who had tried to get shared memory to work on the Mac, and had not been as successful as they had hoped. I hope that is enough homework to warrant an appeal to this group for further pointers. Specifically, 1) Is there any thorough documentation anywhere about how shared memory on the Mac actually works? I need all the background information I can get, and I certainly need to know whether there are hard or soft range limits for the kern.sysv.<whatever> parameters, and what happens if you crowd those limits. Just man pages aren't near enough, and googling and searching XCode documentation hasn't turned up anything interesting yet. 2) What am I likely to break elsewhere in the Mac architecture if I start increasing shared-memory limits? 3) How do the layers of software architecture upon which a Cocoa application runs interact with the shared memory stuff? I haven't tried messing with /etc/rc yet, but in my application, I could not successfully create even a 1 MByte shared-memory segment. I suspect that something else is using enough of the default shared-memory resources that there weren't enough left for my test, and I need to know how much to allow for any such uses. (If this question should go to another group, tell me and I will take it there.) If you are still reading and are curious, what I have is a functioning Lisp system (Wraith Scheme -- see my web site if you are terminally curious (URL below)); I think I have a way to add an access-rights system to it such that multiple copies of the Lisp interpreter can interact with the same heap without stepping on each others' toes. The heap is the big shared memory object I spoke of. (Lisps eat memory as if there were no tomorrow, 1 GByte is not a lot...) Using just separate threads, one for each Lisp interpreter, isn't appropriate, because each interpreter has substantial amounts of truly private data. Just fork (without exec) won't work on a Mac app, at least, not if you want both parent and child to have a functioning GUI. Small numbers of cooperating parallel Lisp processes would be fun, and would be a natural fit to present and likely near-future Mac processor hardware. Enough blathering. Thanks for any hints or suggestions you may have. -- Jay Reynolds Freeman --------------------- Jay_Reynolds_Freeman(a)mac.com http://web.mac.com/jay_reynolds_freeman (personal web site) _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Darwin for mental ray cluster Rendering?
by Ben Byer 13 Jul '07

13 Jul '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Jul 12, 2007, at 10:47 PM, Edward Arrington wrote: _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… There doesn't seem to be a lot of support for ongoing development of Darwin outside of Apple, but hopefully that will change. I'm trying to make a determination about how Maya with mental ray or the mental ray satellite might or might not run on Darwin. I'm interested in using either Maya or just the mental ray satellite app for backend rendering in a small cluster of mixed cpus (PowerPC, Intel, AMD) and I'd rather not have to have one Linux distro for the x86 stuff and one Linux distro for the PowerPC stuff. By all accounts it looks like Dr. Queue will install and run under Darwin as an X11 app and that's the queue management software I want to use. Now I just have to figure out the mental ray part of the puzzle. It seems plausible that mental ray would install on Darwin and maybe be accessible from the command line, since the OS X install probably installs 80% of the application the Darwin component of OS X, but I don't know if that is actually going to work or whether OS X binaries will actually install usable apps on Darwin. PLEASE HELP. I've never even heard for mental ray before now, but a little bit of searching turned this up: http://www.mentalimages.com/2_1_2_configurations/index.html "mental ray® supports all UNIX platforms from Hewlett-Packard (HPPA 2.0, Alpha, IA64), IBM (Power, PowerPC), SGI, Inc. (MIPS, IA64), and Sun Microsystems (Sparc). Also, mental ray supports Windows 2000/XP platforms. Furthermore, mental ray supports LINUX on x86, IA64 Itanium and AMD X86-64 Opteron based platforms and Apple Macintosh OS X (PowerPC and Intel)." Are you trying to install this under OS X, or under Darwin on a generic X86 platform? This email sent to site_archiver(a)lists.apple.com

1 0

Re: Monitoring swapping ...
by Michael Smith 13 Jul '07

13 Jul '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Jul 11, 2007, at 12:04 PM, darwin-dev-request(a)lists.apple.com wrote: I have an application that uses enough memory (Gigabytes) that it sometimes slows way down because of swapping. I'd like to be able to tell the user when that happens, so the user will not give up in disgust. My application is threaded, and the swapping generally takes place in one particular thread, so I could in principle have another thread monitor and report. But I am not familiar enough with low-level Unix memory-management stuff to know whether there is any way to track what is going on, or even where to start to look. Can anyone offer advice or suggestions? (The application's design pattern is model-view-controller. The model is in one thread; it is pretty much straight C++, and that's where the swapping takes place. The view and controller use another thread or two; They are pretty conventional Cocoa/Objective-C/Interface-Builder, and that's where it would be easiest to track swapping and report to the user.) = Mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Rather than attempting to detect one reason why your work thread would be making less forward progress, how about detecting the rate of forward progress that the thread is making? Since this is the thing that you actually care about, it makes more sense to monitor it directly rather than guess at what might be making it slow. Once you have decided what might be an acceptable rate of forward progress, you can caution the user (please, do not use a modal dialog!) that progress is slower than expected and offer suggestions as to how to determine what might be the cause (suggest you advise them on the use of Activity Monitor). This email sent to site_archiver(a)lists.apple.com

1 0

Darwin for mental ray cluster Rendering?
by Edward Arrington 13 Jul '07

13 Jul '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: open() fails to update st_mtime when used with O_TRUNC and 0 byte files
by Mike Frysinger 12 Jul '07

12 Jul '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=t9uJEXGF8XYM1bynOcuWvRprdIptdMB1/FWGwtLh5OPwY8FXoaERERnxyYly9AP8YD/tNrScikF+M6BVSfNTGVgOKOVxBLzU87UnJPDMSg0OM2VZETqZ7OZUVbo/C9wGZzygWN+p2pIhGQNDllQEKhJVF12R34a5kj+dvgs8Fj0= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=mZTvmBYjNWISQBM58+L4PMuZI08muMd+eNIbKFwXlnYxP4iChOSDVcPTkONWph+TOzz+qVwoRUmDNYu69CsrskB/JpIRHHf7BUHm4fiIkD5WToF6MMsMvqf602/7h2JwRPIncWW7KIM7mkajNDKqh09JGoQdsJAN0cTXErNe5I8= On Jul 11, 2007, at 10:19 PM, Mike Frysinger wrote: > i guess it isnt possible to ask for a fix for 10.4.x ... especially > since i dont have any sort of customer $$$ backing the request, just > some lamer who wants to compile the kernel sanely. Fixes to 10.4.x are extremely expensive, particularly at this point in the schedule when everyone is as focused as they are on Leopard. There has to be a significant reward, and not so much in individual customer $$$ as customer numbers who will benefit from it. That's just engineering reality. customer numbers are just a thumb in the wind for how much $$$ will come out of it ;). i honestly dont think the number of people out there doing Linux (kernel) development on an OS X is large enough to even sneeze at let alone commit to. at this point, i'd be happy with just a fix in the default 10.5 setup. -mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On 7/12/07, Jordan K. Hubbard <jkh(a)apple.com> wrote: This email sent to site_archiver(a)lists.apple.com

1 0

Re: open() fails to update st_mtime when used with O_TRUNC and 0 byte files
by Mike Frysinger 12 Jul '07

12 Jul '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=VqrpyF8/XoWohs5KfB6UTA/gunRhVygN48qFuNqAVL/dBBShrUAEswQJOpi4WNPWz8Zo7CpCikqH2KD1YSxjyWnhL+yxasx3fAbXSmZA2MuT7aD9e6jaAL6TdtiWh7Mr5kXMmXiEemvOZC73QZznb9WFP9bEL2bJOWEZ1erIwgU= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=oUQxmKGDcsSBouVzb0jKMWC/Dcz7/l7w2O6zPzQ9MOTDlgFfwDj132U0UJw2aYxSbjALm8q8jhQ7RIaOsj9z+lEEMEpthjRuV1qhz1j8ywduAykWMl+fgWqYv3dCe7pXurE5owk88TObHgHOWaiMMxqkZXP2aR1EbmiKvlR8GaQ= On Jul 11, 2007, at 9:45 PM, Mike Frysinger wrote: > i dont even know what a "DTS representative" ... my guess is you're punting me to the marketing team or something ? "Marketing?" Please don't insult the Developer Tech Support engineers like that. :-) ouch, slap in the face Seriously, DTS is where you go when you need support on development issues beyond what can be provided on a mailing list. You can read more about DTS at the URL in my email signature. i can follow up wherever it takes to get the issue fixed ... the issue i opened in the tracker was closed as a duplicate of another report -- one i cant access. -mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On 7/12/07, Garth Cummings <gcummings(a)apple.com> wrote: This email sent to site_archiver(a)lists.apple.com

1 0