Re: Too many open files error (EMFILE)
Re: Too many open files error (EMFILE)
- Subject: Re: Too many open files error (EMFILE)
- From: Quinn <email@hidden>
- Date: Wed, 2 Mar 2005 16:26:45 +0000
At 16:27 +0100 2/3/05, Jaroslav Snajdr wrote:
the error I get is really EMFILE (errno 24, Too many open files).
OK. Sorry about that. You threw me with the reference to getvnode,
which doesn't raise EMFILE but rather ENFILE.
There are no related kernel messages in any log (console or system).
Indeed. That makes sense given that it can only come from one place,
which is the following code in fdalloc, which doesn't log anything.
if (fdp->fd_nfiles >= lim)
return (EMFILE);
This error means, specifically, that your process has opened too many
files. lim is set using:
lim = min((int)p->p_rlimit[RLIMIT_NOFILE].rlim_cur, maxfiles);
which means that it's the minimum of your process's resource limit or
maxfiles. You can determine maxfiles for your installed kernel using
GDB.
$ gdb /mach_kernel
[...]
(gdb) p (int)maxfiles
$1 = 12288
This is pretty much what you expected.
AFAICT one of N things is happening. Either:
1. your resource limit is lower than you think it is, or
2. maxfiles has been lower is some weird way, or
3. the code that searches the file descriptor array is broken, or
4. the file descriptor array really is that full, but lsof can't see
those files.
The only way I know of distinguishing these cases is to set up
two-machine debugging and poke around the machine with GDB. There
are instructions for setting up two-machine debugging on our web site.
Once you're connected, you can print maxfiles using the command shown
above. That should eliminate case 2. The next step would be to set
up a breakpoint on the line that returns EMFILE and see what's
causing it. You might be able to do that using the Darwin source,
although I generally prefer to do this sort of debugging in assembly.
This is the code from the 10.3.8 kernel.
(gdb) disas fdalloc
Dump of assembler code for function fdalloc:
0x0020ae7c <fdalloc+0>: mflr r0
0x0020ae80 <fdalloc+4>: stmw r21,-44(r1)
0x0020ae84 <fdalloc+8>: stw r0,8(r1)
0x0020ae88 <fdalloc+12>: lis r9,49
0x0020ae8c <fdalloc+16>: stwu r1,-112(r1)
0x0020ae90 <fdalloc+20>: mr r23,r4
0x0020ae94 <fdalloc+24>: lwz r2,20(r3)
0x0020ae98 <fdalloc+28>: mr r21,r5
0x0020ae9c <fdalloc+32>: lwz r24,27568(r9)
0x0020aea0 <fdalloc+36>: lwz r0,132(r2)
0x0020aea4 <fdalloc+40>: lwz r31,12(r3)
0x0020aea8 <fdalloc+44>: cmplw cr7,r24,r0
0x0020aeac <fdalloc+48>: ble- cr7,0x20aeb4 <fdalloc+56>
0x0020aeb0 <fdalloc+52>: mr r24,r0
0x0020aeb4 <fdalloc+56>: lis r22,52
0x0020aeb8 <fdalloc+60>: lwz r8,16(r31)
0x0020aebc <fdalloc+64>: mr r10,r24
0x0020aec0 <fdalloc+68>: cmplw cr7,r24,r8
0x0020aec4 <fdalloc+72>: ble- cr7,0x20aecc <fdalloc+80>
0x0020aec8 <fdalloc+76>: mr r10,r8
0x0020aecc <fdalloc+80>: lhz r0,22(r31)
0x0020aed0 <fdalloc+84>: mr r11,r23
0x0020aed4 <fdalloc+88>: cmpw cr7,r23,r0
0x0020aed8 <fdalloc+92>: bge- cr7,0x20aee0 <fdalloc+100>
0x0020aedc <fdalloc+96>: mr r11,r0
0x0020aee0 <fdalloc+100>: cmpw cr7,r11,r10
0x0020aee4 <fdalloc+104>: lwz r2,0(r31)
0x0020aee8 <fdalloc+108>: lwz r9,4(r31)
0x0020aeec <fdalloc+112>: rlwinm r0,r11,2,0,29
0x0020aef0 <fdalloc+116>: add r3,r2,r0
0x0020aef4 <fdalloc+120>: add r9,r9,r11
0x0020aef8 <fdalloc+124>: bge- cr7,0x20af5c <fdalloc+224>
0x0020aefc <fdalloc+128>: lwz r0,0(r3)
0x0020af00 <fdalloc+132>: cmpwi cr7,r0,0
0x0020af04 <fdalloc+136>: bne- cr7,0x20af48 <fdalloc+204>
0x0020af08 <fdalloc+140>: lbz r0,0(r9)
0x0020af0c <fdalloc+144>: andi. r2,r0,4
0x0020af10 <fdalloc+148>: bne+ 0x20af48 <fdalloc+204>
0x0020af14 <fdalloc+152>: li r0,4
0x0020af18 <fdalloc+156>: stb r0,0(r9)
0x0020af1c <fdalloc+160>: lhz r2,20(r31)
0x0020af20 <fdalloc+164>: cmpw cr7,r11,r2
0x0020af24 <fdalloc+168>: ble- cr7,0x20af2c <fdalloc+176>
0x0020af28 <fdalloc+172>: sth r11,20(r31)
0x0020af2c <fdalloc+176>: lhz r0,22(r31)
0x0020af30 <fdalloc+180>: cmpw cr7,r23,r0
0x0020af34 <fdalloc+184>: bgt- cr7,0x20af3c <fdalloc+192>
0x0020af38 <fdalloc+188>: sth r11,22(r31)
0x0020af3c <fdalloc+192>: stw r11,0(r21)
0x0020af40 <fdalloc+196>: li r3,0
0x0020af44 <fdalloc+200>: b 0x20b030 <fdalloc+436>
0x0020af48 <fdalloc+204>: addi r11,r11,1
0x0020af4c <fdalloc+208>: addi r3,r3,4
0x0020af50 <fdalloc+212>: cmpw cr7,r11,r10
0x0020af54 <fdalloc+216>: addi r9,r9,1
0x0020af58 <fdalloc+220>: blt+ cr7,0x20aefc <fdalloc+128>
0x0020af5c <fdalloc+224>: cmpw cr7,r8,r24
0x0020af60 <fdalloc+228>: li r3,24
0x0020af64 <fdalloc+232>: bge- cr7,0x20b030 <fdalloc+436>
0x0020af68 <fdalloc+236>: cmpwi cr7,r8,49
0x0020af6c <fdalloc+240>: li r26,50
0x0020af70 <fdalloc+244>: ble- cr7,0x20af78 <fdalloc+252>
0x0020af74 <fdalloc+248>: rlwinm r26,r8,1,0,30
0x0020af78 <fdalloc+252>: cmpw cr7,r26,r24
0x0020af7c <fdalloc+256>: ble- cr7,0x20af84 <fdalloc+264>
0x0020af80 <fdalloc+260>: mr r26,r24
0x0020af84 <fdalloc+264>: mulli r29,r26,5
0x0020af88 <fdalloc+268>: li r4,73
0x0020af8c <fdalloc+272>: li r5,0
0x0020af90 <fdalloc+276>: mr r3,r29
0x0020af94 <fdalloc+280>: bl 0x2136a0 <_MALLOC_ZONE>
[...]
0x0020b030 <fdalloc+436>: lwz r0,120(r1)
0x0020b034 <fdalloc+440>: addi r1,r1,112
0x0020b038 <fdalloc+444>: lmw r21,-44(r1)
0x0020b03c <fdalloc+448>: mtlr r0
0x0020b040 <fdalloc+452>: blr
End of assembler dump.
(gdb)
The instruction at 0x0020af60 is the one setting up the error result.
Unfortunately you can't just set a breakpoint on that line because
it's run regardless of whether the error condition is met. Rather,
you should set a breakpoint on 0x0020b030 and break if $r3 is 24.
From memory...
(gdb) b *0x0020b030
(gdb) cond $bpnum $r3==24
If you hit the breakpoint, $r8 is fdp->fd_nfiles and $r24 is lim.
(gdb) p $r8
(gdb) p $r24
These numbers will help you determine if you're hitting case 1.
Beyond that (cases 3 and 4), really weird things are afoot.
S+E
--
Quinn "The Eskimo!" <http://www.apple.com/developer/>
Apple Developer Technical Support * Networking, Communications, Hardware
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Darwin-kernel mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden