July 2010 - Darwin-dev - lists.apple.com

Memory fragmentation in a 32-bit application
by Mitchell J Laurren-Ring 19 Jul '10

19 Jul '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha1; c=relaxed; d=rynosoft.com; h=message-id :from:to:content-type:content-transfer-encoding:mime-version :subject:date; s=rynosoft.com; bh=HJRZv7AJ4wUYrEMBCzZpVFGkbRw=; b=DJSXKitv/pA6ZNFmjIK8r14qlcDPwOtrRGNaU6k0Qxjkvx8xSXybpNuPwUlxS lcBblLdG/BQKh/03uo8qzj2m5gy9duFrywFGAAQwpjF8+kF7eoC1QPbdICZYCP/7 txPOQ5TnSlHSfmuQdiFlzQKVNnsv6zrZR0S6n6ji+ROuQ4= Domainkey-signature: a=rsa-sha1; c=nofws; d=rynosoft.com; h=message-id:from :to:content-type:content-transfer-encoding:mime-version:subject: date; q=dns; s=rynosoft.com; b=OyBtkw8eKano0PCwrb7DOmwiCvyXyY2pu 1S3heZP5W0Li3Y7f7O4/JRV7nV9s2+9jiomvV85yYuzuRPIXL3yHJzRXSNWKePXR KqLMekBmljUf24rSi1Lm6Gv93H91C4mLPo7122HHIgBs6wdaSx2pxK7LhY4ZhKvd /rdA6uI4IM= Hello: -Mick ------ Mitchell J Laurren-Ring lists(a)rynosoft.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I have a 32-bit application that can convert PDF files into Postscript files. We have a customer that processes large batches (100+) of large PDF files (10Mb+). That customer will often experience a crash when our application attempts to allocate memory using malloc which fails and returns nil. The crash occurs because we assert the nil but I'm trying to understand why the failure is occurring. First, I have debugged the problem at the customer site and noted that when the crash occurred our application was not out of VM (we still have around 3Gb available) and there was still tons (over 4Gb of the 8Gb installed) of physical memory available. The OS reports "malloc: *** mmap(size=nnnnnnnn) failed (error code=12)" on the console and presumably returns a nil for the malloc. I cannot confirm this because the code that is being executed at the time is in a library for which I do not have source. The failed allocations can range from 10Mb to 50Mb. I have concluded that the failure is probably happening as result of fragmentation in our VM space but I'm not sure how to confirm that. Are there any tools that can be used to visualize an application's VM space? I've used Instruments quite extensively but it has problems with apps that use a lot of resources and I don't think it has the ability that I seek. This email sent to site_archiver(a)lists.apple.com

1 0

Re: sem_open() returns ENOSPC
by Jonas Maebe 18 Jul '10

18 Jul '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On 18 Jul 2010, at 18:30, Alastair Houghton wrote: > On 18 Jul 2010, at 12:18, Jonas Maebe wrote: > >> The easiest way to solve this is to immediately call sem_unlink() after you've called sem_open(). The semaphore will remain in existence until you call sem_close(). Since all file descriptors are automatically closed when a process terminates (either normally or abnormally), this prevents all semaphore leaking (unless you'd be terminated between the sem_open() and sem_unlink(), but that's very unlikely). > > *However*, if you do this, you can't then obtain the semaphore by name from another process, since the name has been unlinked (it works just like the filesystem in this regard). Since Norm said that he was using them because Darwin does not support unnamed semaphores (sem_init/sem_destroy), that shouldn't be an issue here. Jonas _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: sem_open() returns ENOSPC
by Alastair Houghton 18 Jul '10

18 Jul '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha1; c=relaxed; d=coriolis-systems.com; h= subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; s=aug07; bh=X69RzK/c8uWVegbZqQC8+QREWXg=; b=da1WtlL8deR9Tl20jb5pBp8sVCkQ 8KVZEIS5xOpxjLBhK1bvOZ+4kMfIEuBy3yM1Gd/8iQ+RnuKMLNbFiwLhoVGfFv80 9ScDvfOSWME+68Zz0LmOHcorDBQDd+36I1kueIyG7AUgwk2NcF1htHdfbTZ3XrUR CKy1IeYR7Txlrmk= On 18 Jul 2010, at 12:18, Jonas Maebe wrote: > On 15 Jul 2010, at 01:50, Norm Green wrote: > >> Well that might account for the problem. Since I need (but do not have) unnamed POSIX semaphores in Darwin, I generate a random name for each semaphore but unlink. Sounds like I need to find some way to allocate semaphores with a well known name and then unlink it before attempting to use the name again. > > The easiest way to solve this is to immediately call sem_unlink() after you've called sem_open(). The semaphore will remain in existence until you call sem_close(). Since all file descriptors are automatically closed when a process terminates (either normally or abnormally), this prevents all semaphore leaking (unless you'd be terminated between the sem_open() and sem_unlink(), but that's very unlikely). *However*, if you do this, you can't then obtain the semaphore by name from another process, since the name has been unlinked (it works just like the filesystem in this regard). Kind regards, Alastair. -- http://alastairs-place.net _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: sem_open() returns ENOSPC
by Norm Green 18 Jul '10

18 Jul '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com User-agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.10) Gecko/20100512 Lightning/1.0b1 Thunderbird/3.0.5 Excellent idea Jonas, thanks much. Norm Green On 15 Jul 2010, at 01:50, Norm Green wrote: The easiest way to solve this is to immediately call sem_unlink() after you've called sem_open(). The semaphore will remain in existence until you call sem_close(). Since all file descriptors are automatically closed when a process terminates (either normally or abnormally), this prevents all semaphore leaking (unless you'd be terminated between the sem_open() and sem_unlink(), but that's very unlikely). Jonas _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/normg%40vmware.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On 7/18/2010 4:18 AM, Jonas Maebe wrote: Well that might account for the problem. Since I need (but do not have) unnamed POSIX semaphores in Darwin, I generate a random name for each semaphore but unlink. Sounds like I need to find some way to allocate semaphores with a well known name and then unlink it before attempting to use the name again. This email sent to normg(a)vmware.com This email sent to site_archiver(a)lists.apple.com

1 0

Re: sem_open() returns ENOSPC
by Jonas Maebe 18 Jul '10

18 Jul '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On 15 Jul 2010, at 01:50, Norm Green wrote: > Well that might account for the problem. Since I need (but do not have) unnamed POSIX semaphores in Darwin, I generate a random name for each semaphore but unlink. Sounds like I need to find some way to allocate semaphores with a well known name and then unlink it before attempting to use the name again. The easiest way to solve this is to immediately call sem_unlink() after you've called sem_open(). The semaphore will remain in existence until you call sem_close(). Since all file descriptors are automatically closed when a process terminates (either normally or abnormally), this prevents all semaphore leaking (unless you'd be terminated between the sem_open() and sem_unlink(), but that's very unlikely). Jonas _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: sem_open() returns ENOSPC
by Norm Green 14 Jul '10

14 Jul '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com User-agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.10) Gecko/20100512 Lightning/1.0b1 Thunderbird/3.0.5 Norm Green On 14 Jul 2010, at 23:42, Norm Green wrote: You won't need to call sem_close(), but if you're generating the names for your semaphores algorithmically and you might generate many different names, you *will* need to call sem_unlink(). Semaphores are shared between processes, and once created they continue to exist (with the same name) until they're explicitly deleted with sem_unlink(). Kind regards, Alastair. -- http://alastairs-place.net _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Well that might account for the problem. Since I need (but do not have) unnamed POSIX semaphores in Darwin, I generate a random name for each semaphore but unlink. Sounds like I need to find some way to allocate semaphores with a well known name and then unlink it before attempting to use the name again. On 7/14/2010 3:51 PM, Alastair Houghton wrote: As I said in my post, the process is in start-up when I get the error, so I'm confident I don't have a leak at this point. Do I need to call sem_close() even if the process terminates? This email sent to site_archiver(a)lists.apple.com

1 0

Re: sem_open() returns ENOSPC
by Alastair Houghton 14 Jul '10

14 Jul '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha1; c=relaxed; d=coriolis-systems.com; h= subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; s=aug07; bh=JqLq3Tpjk4JlZTFS6CHwlkEeu1k=; b=g+bRg+hzVT8QGAmSEwniJ52jYt7x NIeui0CyXX4eM/kwMBdLrOk67tVd3RykV9Wl8g6IKafEpZ+Z5azD0kYsr16Fel/5 kezyruatoVpbDw5oD10kQfQIBIcmgBetHF6XL/spdnBcGpvLhtgef+N7oXSa6ez9 yhx9EeIVHBWQG5s= On 14 Jul 2010, at 23:42, Norm Green wrote: > As I said in my post, the process is in start-up when I get the error, so I'm confident I don't have a leak at this point. Do I need to call sem_close() even if the process terminates? You won't need to call sem_close(), but if you're generating the names for your semaphores algorithmically and you might generate many different names, you *will* need to call sem_unlink(). Semaphores are shared between processes, and once created they continue to exist (with the same name) until they're explicitly deleted with sem_unlink(). Kind regards, Alastair. -- http://alastairs-place.net _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: sem_open() returns ENOSPC
by Norm Green 14 Jul '10

14 Jul '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com User-agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.10) Gecko/20100512 Thunderbird/3.0.5 On 14 Jul 2010, at 22:49, Mo McRoberts wrote: On 14-Jul-2010, at 22:44, Norm Green wrote: Hi, ipcs only lists SysV semaphores, whereas the OP is using POSIX semaphores (so it won't help). I'm not sure that there is any equivalent for POSIX semaphores (which is a bit of a shame, actually). Anyway, I agree with your implied diagnosis; the kernel will return ENOSPC if you allocate more than posix_sem_max (default 10,000, configurable via sysctl as kern.posix.sem.max) semaphores. Most likely the OP is leaking semaphores somehow (possibly some sem_close() and sem_unlink() calls are needed somewhere?) Kind regards, Alastair. -- http://alastairs-place.net _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… As I said in my post, the process is in start-up when I get the error, so I'm confident I don't have a leak at this point. Do I need to call sem_close() even if the process terminates? On 7/14/10 3:33 PM, Alastair Houghton wrote: We have a problem where sem_open() returns ENOSPC. This occurs during program startup and repeats until we reboot the machine. Then it goes away for awhile but eventually returns until we reboot the machine again. Is there a better solution than restarting the machine? I'm guessing you've already run 'ipcs' when this problem occurs, and it's not listing lots of semaphores? This email sent to site_archiver(a)lists.apple.com

1 0

Re: sem_open() returns ENOSPC
by Alastair Houghton 14 Jul '10

14 Jul '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha1; c=relaxed; d=coriolis-systems.com; h= subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; s=aug07; bh=YOOoZxf7lWP9xSJuC5YCSA+T2po=; b=nwLGi03PkIgkCvwJtN7x9EOzMrGA 0YqKn7k2mC7Ey8d7U2awmH+uBaJRUuRoG3g7Q/wQbcAxV6d2AbVEfBchHVNvnKvN vG3FTeFQZI6SIALgmcSGhBSppurOstLeJt4+PkY/ymrq+IA2Xj+Oxc3ceWTD8cwY 4JWDexmtci19D/o= On 14 Jul 2010, at 22:49, Mo McRoberts wrote: > On 14-Jul-2010, at 22:44, Norm Green wrote: > >> Hi, >> >> We have a problem where sem_open() returns ENOSPC. This occurs during program startup and repeats until we reboot the machine. Then it goes away for awhile but eventually returns until we reboot the machine again. Is there a better solution than restarting the machine? > > I'm guessing you've already run 'ipcs' when this problem occurs, and it's not listing lots of semaphores? ipcs only lists SysV semaphores, whereas the OP is using POSIX semaphores (so it won't help). I'm not sure that there is any equivalent for POSIX semaphores (which is a bit of a shame, actually). Anyway, I agree with your implied diagnosis; the kernel will return ENOSPC if you allocate more than posix_sem_max (default 10,000, configurable via sysctl as kern.posix.sem.max) semaphores. Most likely the OP is leaking semaphores somehow (possibly some sem_close() and sem_unlink() calls are needed somewhere?) Kind regards, Alastair. -- http://alastairs-place.net _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: sem_open() returns ENOSPC
by Mo McRoberts 14 Jul '10

14 Jul '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On 14-Jul-2010, at 22:44, Norm Green wrote: > Hi, > > We have a problem where sem_open() returns ENOSPC. This occurs during program startup and repeats until we reboot the machine. Then it goes away for awhile but eventually returns until we reboot the machine again. Is there a better solution than restarting the machine? I'm guessing you've already run 'ipcs' when this problem occurs, and it's not listing lots of semaphores? M. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0