October 2006 - Darwin-kernel

mbuf corruption tracking advice
by Andrew Gallatin 23 Oct '06

23 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Does anybody have any advice about how to track down an mbuf data corruption panic? I'm now able to sometimes reproduce mbuf corruption with my driver/hardware on ppc. It seems to happen most frequently after suspend/resume, but sometimes also happens soon after kextload. In my latest panic, I've got a stack that looks like the following. I was wondering what techniques might be recommended to track this down. I've instrumented my driver, and I'm recording ever mbuf I've seen. I cannot find any evidence my driver has ever even touched the mbuf in question. Obviously I must be causing this somehow though.. Thanks, Drew (gdb) bt #0 Debugger (message=0x30cb04 "panic") at /SourceCache/xnu/xnu-792.12.6/osfmk/ppc/model_dep.c:639 #1 0x00026898 in panic (str=0x32c84c "freeing free mbuf") at /SourceCache/xnu/xnu-792.12.6/osfmk/kern/debug.c:202 #2 0x0028c4ec in m_free (m=0x1dc1e200) at /SourceCache/xnu/xnu-792.12.6/bsd/kern/uipc_mbuf.c:600 #3 0x00295558 in sbdroprecord (sb=0x190cb94) at /SourceCache/xnu/xnu-792.12.6/bsd/kern/uipc_socket2.c:1181 #4 0x002922c4 in soreceive (so=0x190cb50, psa=0xcfc3dc8, uio=0x1cb4f00, mp0=0x0, controlp=0x1dbbd000, flagsp=0xcfc3e8c) at /SourceCache/xnu/xnu-792.12.6/bsd/kern/uipc_socket.c:1924 #5 0x00296f38 in recvit (p=0x18ed0f0, s=9, mp=0xcfc3e60, uiop=0x1cb4f00, namelenp=0, retval=0x16c2108) at /SourceCache/xnu/xnu-792.12.6/bsd/kern/uipc_syscalls.c:932 #6 0x0029763c in recvmsg (p=0x18ed0f0, uap=0x16c20c4, retval=0x16c2108) at /SourceCache/xnu/xnu-792.12.6/bsd/kern/uipc_syscalls.c:1255 #7 0x002aae28 in unix_syscall (regs=0x3e0000) at /SourceCache/xnu/xnu-792.12.6/bsd/dev/ppc/systemcalls.c:207 #8 0x000ab930 in noassist () #9 0x3e312e30 in ?? () The mbuf itself looks like this: (gdb) frame 2 #2 0x0028c4ec in m_free (m=0x1dc1e200) at /SourceCache/xnu/xnu-792.12.6/bsd/kern/uipc_mbuf.c:600 600 in /SourceCache/xnu/xnu-792.12.6/bsd/kern/uipc_mbuf.c (gdb) p *m $34 = { m_hdr = { mh_next = 0x1dc1e100, mh_nextpkt = 0x0, mh_len = -1, mh_data = 0xffffffff <Address 0xffffffff out of bounds>, mh_type = 0, mh_flags = -1 }, M_dat = { MH = { MH_pkthdr = { len = -1, rcvif = 0xffffffff, header = 0xffffffff, csum_flags = -1, csum_data = -1, aux = 0xffffffff, vlan_tag = 65535, socket_id = 65535, tags = { slh_first = 0xffffffff } }, MH_dat = { MH_ext = { ext_buf = 0xffffffff <Address 0xffffffff out of bounds>, ext_free = 0xffffffff, ext_size = 4294967295, ext_arg = 0xffffffff <Address 0xffffffff out of bounds>, ext_refs = { forward = 0xffffffff, backward = 0xffffffff } }, MH_databuf = 'y' <repeats 204 times> } }, M_databuf = 'y' <repeats 236 times> } } If I follow the mh_next pointer, the next mbuf looks exactly the same except for the next pointer: (gdb) p *(struct mbuf *)0x1dc1e100 $35 = { m_hdr = { mh_next = 0x1dc1e000, mh_nextpkt = 0xffffffff, <...> And the chain seems to end here: (gdb) p *(struct mbuf *)0x1dc1e000 $36 = { m_hdr = { mh_next = 0x0, mh_nextpkt = 0xffffffff, <....> What would set the data of a bunch of mbufs to all 0xff while preserving their next pointers? Thanks, Drew _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: What, besides open files, can keep a volume busy?
by Quinn 23 Oct '06

23 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Well, yes, by definition. S+E -- Quinn "The Eskimo!" <http://www.apple.com/developer/> Apple Developer Relations, Developer Technical Support, Core OS/Hardware At 10:28 -0700 23/10/06, Rick Mann wrote: Would the open file show up in the VFS data structures? The problem is that "lsof" traces file descriptors from the process down. So it iterates the process list, then iterates the descriptor table within each process that it finds, then works out what those descriptors are. <x-man-page://2/unmount> works in a completely different way [1]. VFS knows which vnodes are associated with which mount points. When you try to unmount a volume, VFS iterates that list reclaiming each vnode (that is, detaching it from the underlying file system node). If this fails, the unmount fails. It typically fails because one of the vnodes has a long term ("use count") reference. A vnode gets a use count reference whenever it's attached to a new descriptor, but it can also get one in other ways (like the file being opened by kernel code). [1] The following is a gross simplification; my apologies to the VFS experts out there. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: What, besides open files, can keep a volume busy?
by Rick Mann 23 Oct '06

23 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Oct 23, 2006, at 09:48 , Quinn wrote: Ah! Okay. Would the open file show up in the VFS data structures? Thanks! -- Rick _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… "lsof" shows open file descriptors. It won't show files that are open by the kernel. It's likely that your disk image software is opening the disk image file from within the kernel, and that's why it's not showing up. This email sent to site_archiver(a)lists.apple.com

1 0

Re: What, besides open files, can keep a volume busy?
by Quinn 23 Oct '06

23 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com S+E -- Quinn "The Eskimo!" <http://www.apple.com/developer/> Apple Developer Relations, Developer Technical Support, Core OS/Hardware _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… At 9:33 -0700 23/10/06, Rick Mann wrote: I'm trying to use lsof in a utility to report processes with open files, and now I'm finding it's unreliable for that. "lsof" shows open file descriptors. It won't show files that are open by the kernel. It's likely that your disk image software is opening the disk image file from within the kernel, and that's why it's not showing up. This email sent to site_archiver(a)lists.apple.com

1 0

Re: What, besides open files, can keep a volume busy?
by Allan Nathanson 23 Oct '06

23 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Make sure that you run lsof as root. sudo lsof +fg /Volumes/... - Allan On Oct 23, 2006, at 12:33 PM, Rick Mann wrote: And looking, I see it on the desktop (it's a bit cluttered). Thanks! -- Rick _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… I've got a network drive mounted, and it shows up in the Finder. I tried to unmount it, but the Finder reports that it's busy. I used lsof to see what processes still had it open, and it reported nothing. I even tried running it as root. I ran it like this: Then I checked the mounts, and got these two (the second is a disk image on the network drive, which is usually unmounted automatically by my backup software): //WORKGROUP;RMANN@TSUNAMINAS/TWINAEROBACK on /Volumes/TWINAEROBACK (nodev, nosuid, mounted by rmann) /dev/disk2s2 on /Volumes/TwinAeroSuperDuperBackup (local, nodev, nosuid, journaled) So, I'm puzzled by lsof not reporting anything. Do I misunderstand its use? Am I missing some flags? I'm trying to use lsof in a utility to report processes with open files, and now I'm finding it's unreliable for that. This email sent to site_archiver(a)lists.apple.com

1 0

What, besides open files, can keep a volume busy?
by Rick Mann 23 Oct '06

23 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com And looking, I see it on the desktop (it's a bit cluttered). Thanks! -- Rick _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… I've got a network drive mounted, and it shows up in the Finder. I tried to unmount it, but the Finder reports that it's busy. I used lsof to see what processes still had it open, and it reported nothing. I even tried running it as root. I ran it like this: Then I checked the mounts, and got these two (the second is a disk image on the network drive, which is usually unmounted automatically by my backup software): //WORKGROUP;RMANN@TSUNAMINAS/TWINAEROBACK on /Volumes/TWINAEROBACK (nodev, nosuid, mounted by rmann) /dev/disk2s2 on /Volumes/TwinAeroSuperDuperBackup (local, nodev, nosuid, journaled) So, I'm puzzled by lsof not reporting anything. Do I misunderstand its use? Am I missing some flags? I'm trying to use lsof in a utility to report processes with open files, and now I'm finding it's unreliable for that. This email sent to site_archiver(a)lists.apple.com

1 0

Re: 4k mbuf alignment on intel?
by Andrew Gallatin 23 Oct '06

23 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Terry Lambert writes: > From past posts, he's already outside KPI-land for his application, > and is OK with rev'ing the code on each SU to be able to do that. Actually, all my drivers are with the KPI. Perhaps you're thinking about my posts where I complain about the restrictions of the KPIs. I'm complaining about them precisely because I'm using them! > On Oct 20, 2006, at 6:03 PM, Josh Graessley wrote: > > It is not possible to maintain your own buffer list and use M_EXT.. > > There is no way to do this using the KPI. If you absolutely need > > this, file a bug report and it may be considered for some future > > release. > > > > -josh > > Thank you for verifying that, Josh. The other big problem with M_EXT buffers is that the last time I checked, IOMbufMemoryCursor was not capable of returning a dma address for anything outside of mcl_paddr[]. This means that private receive buffers allocated by a driver would be hard to forward out another interface.. Drew _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: 4k mbuf alignment on intel?
by Andrew Gallatin 23 Oct '06

23 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Terry Lambert writes: > Sorry, the answer is not that easy, so you're probably going to have > to chase it down; the underlying primitive is vm_allocate_page() from > kernel_memory_allocate(), so it's all page aligned for this usage. That's good news actually, I'd be much happier if the problem is elsewhere, and something that I can fix in the driver without having to appeal to our firmware team to lift the 4KB boundary crossing restriction. > You could check this relatively easily at runtime by ANDing the > returned address with 0x00000fff and making sure it's 0, which I Yes. I've already got that in my development driver version, but no Mac Pro to test on yet :) Thanks for the info! Drew _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: 4k mbuf alignment on intel?
by Terry Lambert 21 Oct '06

21 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com -- Terry -josh On Oct 20, 2006, at 5:54 PM, Terry Lambert wrote: On Oct 20, 2006, at 10:55 AM, Andrew Gallatin wrote: 4KB mbufs has been always been aligned on 4KB boundaries on ppc. Is it true on Intel? If this is not true, how does one allocate one's own external storage in the face of the mbuf kpi? -- Terry Thanks, Drew _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/tlambert%40apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/jgraessley%40apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… >From past posts, he's already outside KPI-land for his application, and is OK with rev'ing the code on each SU to be able to do that. But as Josh states, this is far from recommended, and there's no supported way of doing this, so if it's absolutely necessary (it shouldn't be, even for this case), then filing a bug requesting a supported KPI for it is the way to go. On Oct 20, 2006, at 6:03 PM, Josh Graessley wrote: It is not possible to maintain your own buffer list and use M_EXT.. There is no way to do this using the KPI. If you absolutely need this, file a bug report and it may be considered for some future release. I'm trying to track down some weird mbuf corruption problems with my driver on mac pros, without any access to mac pro hardware. One thing that came to my mind is that I currently assume 4k mbufs as returned by mbuf_allocpacket(..4096..) are aligned on a 4k physical address boundary. If the hardware attempts a DMA across a 4K boundary, it will wrap around and scribble over the first part of the page, so this might explain the mbuf corruption. Sorry, the answer is not that easy, so you're probably going to have to chase it down; the underlying primitive is vm_allocate_page () from kernel_memory_allocate(), so it's all page aligned for this usage. You could check this relatively easily at runtime by ANDing the returned address with 0x00000fff and making sure it's 0, which I recommend if you want to reassure yourself that that's the case; however, I think you are going to have to look elsewhere for your problem. It's possible to maintain your own buffer list and use M_EXT; there are several drivers that do this internally, if they have a limited ability to address memory (e.g. if they don't implement all of the address lines, don't hook them all up, or are over a bus that doesn't support it (e.g. using a 32 bit device on a machine with more than 4G of memory, where the DMA target would be above the 4G physical memory boundary). I'm not sure if there are any public examples of drivers using this, but it's possible to do. In general, though, you will end up taking an extra copy (e.g. for pull up) if you end up triggering that code path, so you should avoid this unless you don't really have a choice in the matter. If you need to do this, FreeBSD does it exactly this same way as well (e.g. that's how "sendfile" works in FreeBSD, and I believe there are some network drivers over there that use it because of bus limits, too). This email sent to tlambert(a)apple.com This email sent to jgraessley(a)apple.com This email sent to site_archiver(a)lists.apple.com

1 0

Re: 4k mbuf alignment on intel?
by Josh Graessley 21 Oct '06

21 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com -josh On Oct 20, 2006, at 5:54 PM, Terry Lambert wrote: On Oct 20, 2006, at 10:55 AM, Andrew Gallatin wrote: 4KB mbufs has been always been aligned on 4KB boundaries on ppc. Is it true on Intel? If this is not true, how does one allocate one's own external storage in the face of the mbuf kpi? -- Terry Thanks, Drew _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com It is not possible to maintain your own buffer list and use M_EXT.. There is no way to do this using the KPI. If you absolutely need this, file a bug report and it may be considered for some future release. I'm trying to track down some weird mbuf corruption problems with my driver on mac pros, without any access to mac pro hardware. One thing that came to my mind is that I currently assume 4k mbufs as returned by mbuf_allocpacket(..4096..) are aligned on a 4k physical address boundary. If the hardware attempts a DMA across a 4K boundary, it will wrap around and scribble over the first part of the page, so this might explain the mbuf corruption. Sorry, the answer is not that easy, so you're probably going to have to chase it down; the underlying primitive is vm_allocate_page () from kernel_memory_allocate(), so it's all page aligned for this usage. You could check this relatively easily at runtime by ANDing the returned address with 0x00000fff and making sure it's 0, which I recommend if you want to reassure yourself that that's the case; however, I think you are going to have to look elsewhere for your problem. It's possible to maintain your own buffer list and use M_EXT; there are several drivers that do this internally, if they have a limited ability to address memory (e.g. if they don't implement all of the address lines, don't hook them all up, or are over a bus that doesn't support it (e.g. using a 32 bit device on a machine with more than 4G of memory, where the DMA target would be above the 4G physical memory boundary). I'm not sure if there are any public examples of drivers using this, but it's possible to do. In general, though, you will end up taking an extra copy (e.g. for pull up) if you end up triggering that code path, so you should avoid this unless you don't really have a choice in the matter. If you need to do this, FreeBSD does it exactly this same way as well (e.g. that's how "sendfile" works in FreeBSD, and I believe there are some network drivers over there that use it because of bus limits, too). _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/tlambert% 40apple.com This email sent to tlambert(a)apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/jgraessley% 40apple.com This email sent to jgraessley(a)apple.com smime.p7s

1 0