Re: Troublesome app when scaling it up, how to proceed?
Re: Troublesome app when scaling it up, how to proceed?
- Subject: Re: Troublesome app when scaling it up, how to proceed?
- From: "Gerben Wierda" <email@hidden>
- Date: Thu, 25 Oct 2007 16:49:27 +0200 (CEST)
- Importance: Normal
> Trying to simply this to the known facts:
>
> * Your app is running out of file-handles and failing. You know this
> because you modified the limit and it stops failing. Correct?
Correct.
> On 25 Oct 2007, at 10:28, Gerben Wierda wrote:
>
>> I cannot use DO for communication because I want to communicate
>> with the
>> stdin, stdout, stderr of regular unix cli tools like file, tar,
>> openssl,
>> gpg, perl, sh, etc.
>
> * You're running a lot of sub-processes and connecting to them with
> pipes. At least 3 per process (stdout, stdin, stderr) and possibly
> more (+named ones for privileged processes).
>
>> If there would be an NSPrivilegedTask it would become a lot easier.
>
> It might simplify your code, but it wouldn't solve your problem. That
> routine would still use file-handles to communicate. This is, I
> think, a red herring. If you are talking to stdin/out/err these are
> pipes however you connect to them, using Cocoa or not.
It might be that the way I do connect to them, release my objects etc,
keeps them around longer than expected.
> So the question is 'why aren't they closing?'
Exactly. Or *maybe*, why are they closing (as I *think* I close
everything) but are they not returned to the system?
> In this situation I would be using lsof to monitor the open file
> handles as the app runs. Something like
>
> lsof -p <pid> -n +r 5
>
> will show you all the open files of your process, and which ones are
> lasting longer than they should. Alternatively, using sharks 'System
> Trace' may help by allowing you to see what file-handles are used by
> what system call, and when they should have been released. That's not
> as easy to decode though.
I know lsof (I think I used lsof the last time to see if it could help me
find out what was happening) I'll look into Shark.
> I seem to remember something about pipes or sockets needing to be
> closed from both ends for them to be released, and the process just
> dying isn't enough. I've seen 'zombie' pipes hang around before (and
> get cleaned up after a while) and I think it was something like this.
> (This is from general Unix programming, nothing Mac specific)
I recall something like that as well. I suspect the result will be that if
they are zombie pipes they will not show up with lsof -p.
Suppose I run tar like this
- my app creates named pipes a, b, c, d and e
- my app starts reading from b and c in separate threads (these report the
results back to my main thread using DO)
- my app starts "helper tar"
- helper connects stdin, stdout and stderr to named pipes a, b and c,
reads the environment from d and reports pid back over e. d and e are
closed when helper is done with them.
- as soon as the pid has been reported, my app writes environment over to
helper
- d and e are closed by my app as soon as my app is done with them
- as soon as environment has been received, helper closes d
- helper now execs tar, which inherits environment and stdin, stdout and
stderr which are connected to the named pipes a, b and c.
- tar completes and dies
could zombie pipes be created when apps like tar do not explicitely close
stdout/stderr before exiting?
G
PS. I am aware that one can have two different discussions here. One about
if this setup is at all the only way to do this and one that *if* you want
to do it in this convoluted and complex manner. I'd like to discuss the
second one now, not because I like my setup (I do not), but because I
haven't been able to find a safe way to do it the preferred way when using
a *generalized* helper. I would like to get rid of this, but while I will
be investigating that angle I would like to solve my current riddle, if
only out of technical curiosity.
_______________________________________________
Cocoa-dev mailing list (email@hidden)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden