site_archiver@lists.apple.com Delivered-To: darwin-kernel@lists.apple.com Hello, recently I tried to run fsstress file system stress test tool against HFS. It turned out that it's quite easy to crash XNU (517.7.7) by doing this. First, I am getting stack overflow. By deciphering paniclog from gdb, following stack trace maybe reconstructed: hfs_fsync+1312 0x7c721b79 vinvalbuf+116 0x2f9c0000 vclean+836 0x418e002c vgonel+108 0x801f0014 vrecycle+36 0x38000001 hfs_inactive+1052 0x800100d8 vrele+312 0x805f0020 [clean_up_name_parent_ptrs()] vclean+1252 0x7fc3f378 vgonel+108 0x801f0014 vrecycle+36 0x38000001 hfs_inactive+1052 0x800100d8 vrele+312 0x805f0020 [clean_up_name_parent_ptrs()] vclean+1252 0x7fc3f378 vgonel+108 0x801f0014 vrecycle+36 0x38000001 hfs_inactive+1052 0x800100d8 ... clean_up_name_parent_ptrs() is not in the stack dump, but vclean+1252 is instruction next to the call to clean_up_name_parent_ptrs(): 0x000c1e98 <vclean+1248>: bl 0xc1034 <clean_up_name_parent_ptrs> 0x000c1e9c <vclean+1252>: mr r3,r30 These recursive calls to VOP_INACTIVE() looks flaky. Note that fsstress is known to create large and deep directory hierarchies, and it looks like in the worst case vclean() traverses file system up to the root recursively---sure recipe for disaster. At the lucky times when fsstress doesn't crashes with stack overflow, it hangs: #3 0x0002b300 in thread_block_reason (continuation=0, reason=0) at osfmk/kern/sched_prim.c:1660 #4 0x00069520 in vm_object_terminate (object=0x1c93960) at osfmk/vm/vm_object.c:1036 #5 0x00068f88 in vm_object_deallocate (object=0x1c93960) at osfmk/vm/vm_object.c:784 #6 0x0006c46c in vm_object_release_name (object=0x1c93968, flags=2) at osfmk/vm/vm_object.c:4451 #7 0x00229bd4 in ubc_release_named (vp=0x1ca5690) at bsd/kern/ubc_subr.c:1002 #8 0x00228ffc in ubc_uncache (vp=0x1ca5690) at bsd/kern/ubc_subr.c:452 #9 0x001eba68 in hfs_removefile (dvp=0x1ca8c00, vp=0x320000, cnp=0x14debe44, options=0) at bsd/hfs/hfs_vnops.c:2071 #10 0x000c704c in _unlink (p=0x19bfdc8, uap=0x1ca5690, retval=0x2b1a4, nodelbusy=603988008) at bsd/sys/vnode_if.h:620 #11 0x00243c24 in unix_syscall (regs=0x1890764) at bsd/dev/ppc/systemcalls.c:156 object->paging_in_progress it waits for is interesting: (gdb) p *object $1 = { ... paging_in_progress = 4294967294, ... } The number any self-respecting kernel developer has dreams about. :) Which leads one to ask two questions: - where unbalanced ->paging_in_progress decrement is done? And - why wakeup wasn't performed when ->paging_in_progress reached 0? Looking at all places where ->paging_in_progress is modified I see interesting thing: vm_object_paging_end() correctly does a wakeup when paging counter drops to 0. But UNLOCK_THINGS macro in vm_fault.c doesn't. Another possible reason of missed wakeup is following fragment from vm_fault_page(): vm_object_paging_end(object); vm_object_collapse(object, offset); vm_object_paging_begin(object); that comment above it aptly describes as "ugly". Suppose object->paging_in_progress was 0 initially, vm_object_paging_end() decrements it to -1. If some other thread calls vm_object_paging_wait() at this moment, it will go to sleep... to infinite sleep, because vm_object_paging_begin() won't wake it up (except there is some serialization mechanism that protects this from happening). fstress sources can be found at http://cvs.sourceforge.net/viewcvs.py/ltp/ltp/testcases/kernel/fs/fsstress/ I had to made some trivial fixes to compile in on XNU. Run it as fsstress -d . -n 10000 -p 1 _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.a... This email sent to site_archiver@lists.apple.com