fsstress testing.
fsstress testing.
- Subject: fsstress testing.
- From: Nikita Danilov <email@hidden>
- Date: Mon, 28 Mar 2005 21:08:05 +0400
Hello,
recently I tried to run fsstress file system stress test tool against
HFS. It turned out that it's quite easy to crash XNU (517.7.7) by doing
this.
First, I am getting stack overflow. By deciphering paniclog from gdb,
following stack trace maybe reconstructed:
hfs_fsync+1312 0x7c721b79
vinvalbuf+116 0x2f9c0000
vclean+836 0x418e002c
vgonel+108 0x801f0014
vrecycle+36 0x38000001
hfs_inactive+1052 0x800100d8
vrele+312 0x805f0020
[clean_up_name_parent_ptrs()]
vclean+1252 0x7fc3f378
vgonel+108 0x801f0014
vrecycle+36 0x38000001
hfs_inactive+1052 0x800100d8
vrele+312 0x805f0020
[clean_up_name_parent_ptrs()]
vclean+1252 0x7fc3f378
vgonel+108 0x801f0014
vrecycle+36 0x38000001
hfs_inactive+1052 0x800100d8
...
clean_up_name_parent_ptrs() is not in the stack dump, but vclean+1252 is
instruction next to the call to clean_up_name_parent_ptrs():
0x000c1e98 <vclean+1248>: bl 0xc1034 <clean_up_name_parent_ptrs>
0x000c1e9c <vclean+1252>: mr r3,r30
These recursive calls to VOP_INACTIVE() looks flaky. Note that fsstress
is known to create large and deep directory hierarchies, and it looks
like in the worst case vclean() traverses file system up to the root
recursively---sure recipe for disaster.
At the lucky times when fsstress doesn't crashes with stack overflow, it
hangs:
#3 0x0002b300 in thread_block_reason (continuation=0, reason=0) at osfmk/kern/sched_prim.c:1660
#4 0x00069520 in vm_object_terminate (object=0x1c93960) at osfmk/vm/vm_object.c:1036
#5 0x00068f88 in vm_object_deallocate (object=0x1c93960) at osfmk/vm/vm_object.c:784
#6 0x0006c46c in vm_object_release_name (object=0x1c93968, flags=2) at osfmk/vm/vm_object.c:4451
#7 0x00229bd4 in ubc_release_named (vp=0x1ca5690) at bsd/kern/ubc_subr.c:1002
#8 0x00228ffc in ubc_uncache (vp=0x1ca5690) at bsd/kern/ubc_subr.c:452
#9 0x001eba68 in hfs_removefile (dvp=0x1ca8c00, vp=0x320000, cnp=0x14debe44, options=0) at bsd/hfs/hfs_vnops.c:2071
#10 0x000c704c in _unlink (p=0x19bfdc8, uap=0x1ca5690, retval=0x2b1a4, nodelbusy=603988008) at bsd/sys/vnode_if.h:620
#11 0x00243c24 in unix_syscall (regs=0x1890764) at bsd/dev/ppc/systemcalls.c:156
object->paging_in_progress it waits for is interesting:
(gdb) p *object
$1 = {
...
paging_in_progress = 4294967294,
...
}
The number any self-respecting kernel developer has dreams about. :)
Which leads one to ask two questions:
- where unbalanced ->paging_in_progress decrement is done? And
- why wakeup wasn't performed when ->paging_in_progress reached 0?
Looking at all places where ->paging_in_progress is modified I see
interesting thing: vm_object_paging_end() correctly does a wakeup when
paging counter drops to 0. But UNLOCK_THINGS macro in vm_fault.c
doesn't.
Another possible reason of missed wakeup is following fragment from
vm_fault_page():
vm_object_paging_end(object);
vm_object_collapse(object, offset);
vm_object_paging_begin(object);
that comment above it aptly describes as "ugly". Suppose
object->paging_in_progress was 0 initially, vm_object_paging_end()
decrements it to -1. If some other thread calls vm_object_paging_wait()
at this moment, it will go to sleep... to infinite sleep, because
vm_object_paging_begin() won't wake it up (except there is some
serialization mechanism that protects this from happening).
fstress sources can be found at
http://cvs.sourceforge.net/viewcvs.py/ltp/ltp/testcases/kernel/fs/fsstress/
I had to made some trivial fixes to compile in on XNU. Run it as
fsstress -d . -n 10000 -p 1
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Darwin-kernel mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden