Re: Crash with FD_SET when there are more than 1024 open files
Re: Crash with FD_SET when there are more than 1024 open files
- Subject: Re: Crash with FD_SET when there are more than 1024 open files
- From: Joel Reymont <email@hidden>
- Date: Thu, 15 Apr 2010 17:00:41 +0100
I used to routinely open about 50K sockets.
To accomplish this I edited __DARWIN_FD_SETSIZE in /usr/include/sys/_structs.h to set it to a higher number in addition to the following
/etc/sysctl.conf
kern.maxfiles=102400
kern.maxfilesperproc=40960
net.inet.ip.portrange.hifirst=10000
net.inet.ip.portrange.first=10000
What I found is that Mac OSX was far slower than Linux in this scenario. Search for my name in the archives of this list to read through the lively discussion(s).
The issue you may have is that Bonjour was compiled with FD_SETSIZE of 1024.
On Apr 15, 2010, at 4:54 PM, Stefan Haller wrote:
> We are investigating a reproducible crash that happens in Bonjour code
> when we have many files open. The crash happens in
> DNSServiceProcessResult (part of mDNSResponder), which contains the
> following code:
>
> static int more_bytes(dnssd_sock_t sd)
> {
> struct timeval tv = { 0, 0 };
> fd_set readfds;
> FD_ZERO(&readfds);
> FD_SET(sd, &readfds);
> return(select(sd+1, &readfds, (fd_set*)NULL, (fd_set*)NULL, &tv) > 0);
> }
>
> If sd is greater than 1024, bad things happen; in particular, if sd
> happens to be 1669, you get an immediate crash on return from the
> function because the code overwrites the saved ebp register.
>
> See also <rdar://problem/7701369> to which I attached a small example
> program that demonstrates the crash.
>
> Now, this makes me wonder whether it is ever possible, for any
> application, to exceed the number of 1024 open files. Even if Apple
> fixes that particular piece of code (for example by defining FD_SETSIZE
> to a greater value before including <sys/types.h>), other code which we
> can't control could have the same problem. That could be OS code,
> third-party libraries that we link against, plugins that we load.
>
> One possible workaround would be to somehow ensure that there's always a
> free range of file descriptors below 1024 available for code other than
> ours. I wouldn't know how to do that though; any suggestions?
>
> (We made a quick test to open a hundred dummy files before opening our
> real files, and close them right before calling the offending OS
> function, in the hope that that the Bonjour code would then use these;
> this didn't help though.)
>
> Thanks,
> Stefan
>
>
> --
> Stefan Haller
> Berlin, Germany
> http://www.haller-berlin.de/
> _______________________________________________
> Do not post admin requests to the list. They will be ignored.
> Darwin-dev mailing list (email@hidden)
> Help/Unsubscribe/Update your Subscription:
>
> This email sent to email@hidden
---
http://es.linkedin.com/in/joelreymont
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Darwin-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden