Re: vm_protect with VM_PROT_COPY not working on Leopard

Hi,

I'm trying to patch the malloc function, and my solution works under 10.4 but not 10.5. As I understand it, I need to call vm_protect() to get write access to the malloc function. It seems that malloc is in a system library which is shared across all processes, so I need the VM_PROT_COPY flag to enable copy-on-write semantics. Here's the code I'm using:

int retval= vm_protect( mach_task_self(), reinterpret_cast<vm_address_t>(fnMalloc),
 MALLOC_HEADER_BYTES, false, VM_PROT_READ | VM_PROT_COPY );
...
// Patch the malloc function with a jmp instruction
fnMalloc[0]= 0xE9;
*reinterpret_cast<int32*>(fnMalloc+1)= reinterpret_cast<int32>(pStubFunction) - reinterpret_cast<int32>(fnMalloc) - 5;
...

I'm getting a segfault when I attempt to actually patch the malloc function under 10.5, but this works fine under 10.4. Does anyone why this might be failing, or have an idea about how I can patch the malloc function successfully under Leopard?

In case anyone is wondering, I need to do this as part of our development process for audio plug-ins. These plug-ins should not call certain functions (including malloc) from the high-priority processing thread to avoid priority inversion and related problems. We have special debug builds which patch the malloc function, and if a malloc is detected in the processing thread, we get a very nice call stack in the debugger which makes it trivial to find the source of these problems.

I've looked at other solutions for patching functions, and actually writing to the code page seems like the cleanest way to go. This code is only for internal use so we don't even need to worry about reverting the system to its original state, etc., as long as this helps us track down problems.

Note also that a previous post addressed this issue:
http://lists.apple.com/archives/darwin-kernel/2005/Feb/msg00045.html
but I didn't have any luck with the DYLD_SUFFIX solution, and the copy-on-write solution given there is what I'm posting about.

Anyway I hope this is clear and any assistance would be most appreciated.

Regards,
Jeremy

_______________________________________________
Do not post admin requests to the list. They will be ignored.
Darwin-dev mailing list      (email@hidden)
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/darwin-dev/email@hidden

This email sent to email@hidden