-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 - -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thursday, March 20, 2003, at 10:50 AM, Stockhoff, Cory wrote:

Hello Everybody,

I apologize for the length of this post, but I hoped to provide enough

info to give somebody an "aha!" moment.

Aha! :)

Our product consists of user and kernel components implementing a VPN

client. The kernel component is a KEXT that processes inbound and

outbound

IP packets intercepted by a DLIL. The DLIL is pushed onto every

network

device that becomes active.

The crash we are seeing is a kernel panic that occurs only on

multiple-CPU

hosts, and is infrequent. Our KEXT tries to allocate some kernel

memory by

calling "kmem_alloc()". The panic message is always "panic(cpu [0 or

1]):

thread_invoke: preemption_level 1". The most reliable way to

duplicate the

problem is to engage the software (i.e. connect to a VPN switch via an

encrypted VPN tunnel), mount a remote shared volume (AppleTalk over

IP), and

do heavy reads and writes to the volume.

Our customers have seen the problem on both Jaguar (10.2.x) and Puma

(10.1.x) OS's. Our testing and debugging has been on Jaguar hosts,

10.2.3

(Darwin 6.3). Our application is built on Puma.

Various checks have been tried, none successful:

1) Don't call kmem_alloc() if preemption level is 1. It turns out

that

the preemption level is often 1 during normal operation, and is

usually not a problem. However, not returning allocated memory

every time preemption level is 1 does cause problems for our

software.

If you are holding a spinlock or, you are allocating from a network callback thread, then you may see this crash when allocating memory. kmem_alloc can block and blocking is a big no-no when holding a spinlock or when interrupts are disabled. If alloc does block, then you will get this panic, if it doesn't block then you won't panic. Unfortunately, there is no easy way to tell if an allocation will cause alloc to block. To get around this problem, I allocated a reserve pool at kext load time and at allocation time if interrupts are disabled and kalloc_noblock() returns nil, I grab a block from this reserve. This worked for me in this particular case, because we always disable interrupts when grabbing a spinlock. Here is some pseudo code if (interrupts enabled) { mem = kalloc(size); } else { mem = kalloc_noblock(size); if (!mem) mem = my_reserve_alloc(); } return (mem);

The stack trace can vary a bit, but this one is typical (and from

Jaguar):

0x000856cc print_backtrace+176

0x00085afc Debugger+108

0x000287a8 panic+488

0x00033eec thread_invoke+72

0x000344d0 thread_block_reason+212

0x0008d51c mlInUse+16

0x00060b60 vm_fault_wire_fast+284

0x00064f88 vm_map_wire_nested+2988

0x000651a0 vm_map_wire+120

0x00061a78 kernel_memory_allocate+600

You can see the problem in the backtrace, the thread tries to block because the allocation could not be immediately filled (thread_block_reason). BTW, this problem can happen on single CPU machines too, but is much more likely to happen on dualies. HTH. Brian Bergstrand <http://www.classicalguitar.net/brian/> PGP Key ID: 0xB6C7B6A2 It is easy to convert a bug to a feature; document it. - The Microsoft Employee Handbook, p.215, section B, article VII - -----BEGIN PGP SIGNATURE----- Version: PGP 8.0 iQA/AwUBPnnuwXnR2Fu2x7aiEQIpAACfZkR8pRqVjeQniPIv2Ynjh/IV5+4AoLPO S/GvkjmAqouvnX+GDOTQebJ/ =iXu+ - -----END PGP SIGNATURE----- Brian Bergstrand <http://www.classicalguitar.net/brian/> PGP Key ID: 0xB6C7B6A2 Genius may have its limitations, but stupidity is not thus handicapped. - - Elbert Hubbard -----BEGIN PGP SIGNATURE----- Version: PGP 8.0 iQA/AwUBPnnu+3nR2Fu2x7aiEQIbDACfVgmMfGfXWvZdALXDuYaz4AK88XUAoNia O84epg7QHYQUZgxzwSA3avXG =nwFc -----END PGP SIGNATURE----- _______________________________________________ darwin-kernel mailing list | darwin-kernel@lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.