Re: Dual-CPU Kernel Panic
Re: Dual-CPU Kernel Panic
- Subject: Re: Dual-CPU Kernel Panic
- From: Brian Bergstrand <email@hidden>
- Date: Thu, 20 Mar 2003 11:40:25 -0600
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
- -----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Thursday, March 20, 2003, at 10:50 AM, Stockhoff, Cory wrote:
>
Hello Everybody,
>
>
I apologize for the length of this post, but I hoped to provide enough
>
info to give somebody an "aha!" moment.
>
Aha! :)
>
Our product consists of user and kernel components implementing a VPN
>
client. The kernel component is a KEXT that processes inbound and
>
outbound
>
IP packets intercepted by a DLIL. The DLIL is pushed onto every
>
network
>
device that becomes active.
>
>
The crash we are seeing is a kernel panic that occurs only on
>
multiple-CPU
>
hosts, and is infrequent. Our KEXT tries to allocate some kernel
>
memory by
>
calling "kmem_alloc()". The panic message is always "panic(cpu [0 or
>
1]):
>
thread_invoke: preemption_level 1". The most reliable way to
>
duplicate the
>
problem is to engage the software (i.e. connect to a VPN switch via an
>
encrypted VPN tunnel), mount a remote shared volume (AppleTalk over
>
IP), and
>
do heavy reads and writes to the volume.
>
>
Our customers have seen the problem on both Jaguar (10.2.x) and Puma
>
(10.1.x) OS's. Our testing and debugging has been on Jaguar hosts,
>
10.2.3
>
(Darwin 6.3). Our application is built on Puma.
>
>
Various checks have been tried, none successful:
>
1) Don't call kmem_alloc() if preemption level is 1. It turns out
>
that
>
the preemption level is often 1 during normal operation, and is
>
usually not a problem. However, not returning allocated memory
>
every time preemption level is 1 does cause problems for our
>
software.
>
If you are holding a spinlock or, you are allocating from a network
callback thread, then you may see this crash when allocating memory.
kmem_alloc can block and blocking is a big no-no when holding a
spinlock or when interrupts are disabled. If alloc does block, then you
will get this panic, if it doesn't block then you won't panic.
Unfortunately, there is no easy way to tell if an allocation will cause
alloc to block.
To get around this problem, I allocated a reserve pool at kext load
time and at allocation time if interrupts are disabled and
kalloc_noblock() returns nil, I grab a block from this reserve.
This worked for me in this particular case, because we always disable
interrupts when grabbing a spinlock.
Here is some pseudo code
if (interrupts enabled) {
mem = kalloc(size);
} else {
mem = kalloc_noblock(size);
if (!mem)
mem = my_reserve_alloc();
}
return (mem);
>
>
The stack trace can vary a bit, but this one is typical (and from
>
Jaguar):
>
>
0x000856cc print_backtrace+176
>
0x00085afc Debugger+108
>
0x000287a8 panic+488
>
0x00033eec thread_invoke+72
>
0x000344d0 thread_block_reason+212
>
0x0008d51c mlInUse+16
>
0x00060b60 vm_fault_wire_fast+284
>
0x00064f88 vm_map_wire_nested+2988
>
0x000651a0 vm_map_wire+120
>
0x00061a78 kernel_memory_allocate+600
>
You can see the problem in the backtrace, the thread tries to block
because the allocation could not be immediately filled
(thread_block_reason).
BTW, this problem can happen on single CPU machines too, but is much
more likely to happen on dualies.
HTH.
Brian Bergstrand
<
http://www.classicalguitar.net/brian/> PGP Key ID: 0xB6C7B6A2
It is easy to convert a bug to a feature; document it. - The Microsoft
Employee Handbook, p.215, section B, article VII
- -----BEGIN PGP SIGNATURE-----
Version: PGP 8.0
iQA/AwUBPnnuwXnR2Fu2x7aiEQIpAACfZkR8pRqVjeQniPIv2Ynjh/IV5+4AoLPO
S/GvkjmAqouvnX+GDOTQebJ/
=iXu+
- -----END PGP SIGNATURE-----
Brian Bergstrand
<
http://www.classicalguitar.net/brian/> PGP Key ID: 0xB6C7B6A2
Genius may have its limitations, but stupidity is not thus handicapped.
- - Elbert Hubbard
-----BEGIN PGP SIGNATURE-----
Version: PGP 8.0
iQA/AwUBPnnu+3nR2Fu2x7aiEQIbDACfVgmMfGfXWvZdALXDuYaz4AK88XUAoNia
O84epg7QHYQUZgxzwSA3avXG
=nwFc
-----END PGP SIGNATURE-----
_______________________________________________
darwin-kernel mailing list | email@hidden
Help/Unsubscribe/Archives:
http://www.lists.apple.com/mailman/listinfo/darwin-kernel
Do not post admin requests to the list. They will be ignored.