• Open Menu Close Menu
  • Apple
  • Shopping Bag
  • Apple
  • Mac
  • iPad
  • iPhone
  • Watch
  • TV
  • Music
  • Support
  • Search apple.com
  • Shopping Bag
 

Lists

Open Menu Close Menu
  • Terms and Conditions
  • Lists hosted on this site
  • Email the Postmaster
  • Tips for posting to public mailing lists
Re: Kernel extension debugging tool?
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Kernel extension debugging tool?


  • Subject: Re: Kernel extension debugging tool?
  • From: Rob Barris <email@hidden>
  • Date: Tue, 15 Nov 2005 20:41:56 -0800


On Nov 15, 2005, at 12:03 PM, Sam Vaughan wrote:

On 14/11/2005, at 7:03 AM, Daniel Azuelos <email@hidden> wrote:



What's the best way, method to debug a kernel extension?

I've a slow memory leak to locate precisely and l can't wait the many

hours it needs to lead to a real system problem.



For memory leaks I'd recommend adding some memory debugging code to  

your kernel extension.  How far you take it depends on how complex  

your kernel extension is and therefore whether you think you'll get  

enough return on the investment.  I recently had a slow mbuf leak  

that prompted me to beef up our memory debugging code, and it's now  

at a point where I can monitor outstanding allocations on a live  

system grouped by the symbolic backtraces of all the allocation paths  

in the code.  This makes memory leaks stand out like a sore thumb!


Firstly, you'll need to funnel all your memory allocations and frees  

through the same functions so you've only got one place to add  

debugging code that will catch everything.  I changed our code to use  

kernel memory allocation for mbufs to make sure I could catch mbuf  

misuse as well.


Next, change the code to allocate more space than the original  

request, allowing you to store extra information like the current  

backtrace and the size of the allocation.  At this point you might  

want to add space to both ends and poison it with magic values like  

0xdefec8ed or something similarly childish. ;o)  Poisoning memory  

after allocating it and after freeing it also helps trap code that  

reuses freed memory.  It'd be even nicer to have read-only guard  

pages to generate VM exceptions on writes outside the bounds of  

allocated memory, but putting an entire page on either side of small  

allocations is going to fill up some of the kernel's zones very quickly.


Add each new allocation's address to a hash table, and remove it  

again when it is freed.  Verify that your free calls specify the same  

size as the original allocation.  You can use this hash table at  

unload/quiesce time to check for missing frees.


Maintain a list of unique allocation backtraces.  I just store four  

addresses for each backtrace since going further back than that often  

takes you outside your kext and into the kernel, giving you unique  

backtraces where you would consider them to be the same.  Alongside  

the unique backtrace entries, you can maintain allocation and free  

counts, total amount allocated, etc.


    The "paranoid malloc" we use in game debugging does all that stuff and adds one more wrinkle: freed memory blocks are immediately scrubbed with a fill pattern but those blocks are put on a free list and not allowed to be recycled for any other allocation til they cool off a bit (seconds to minutes, depending how much bloat you can tolerate and how much RAM you have).
    We have a watchdog that runs around all the freed blocks periodically to make sure they are still scrubbed - if one isn't scrubbed any more, it barks about it and dumps out the "who allocated" and "who freed" call stacks, as well as showing which offsets went sour.  Conversely any faulty code that reads from freed blocks usually fails spectacularly enough that it's easy to spot (regs full of F1F1F1F1 etc).  We call that "lazy free".  The same watchdog also scans the live block list to make sure no block-end has been overrun etc.

    Hashing call stacks / stack-crawls is very useful - we use 8 or 16 deep crawls.  So the meta data we have to keep for each live block is just a uint for the allocator-path and the freer-path, an index into the stack-crawl hash table.  That said we only invoke this tool on machines with 2GB or more RAM handy.

Rob

 _______________________________________________
Do not post admin requests to the list. They will be ignored.
Macnetworkprog mailing list      (email@hidden)
Help/Unsubscribe/Update your Subscription:

This email sent to email@hidden

  • Prev by Date: Re: Kernel extension debugging tool?
  • Next by Date: Re: Kernel extension debugging tool?
  • Previous by thread: Re: Kernel extension debugging tool?
  • Next by thread: Re: Kernel extension debugging tool?
  • Index(es):
    • Date
    • Thread