I have a launchd job that runs a little shell bash script for some
rsync backup action. I'm running 10.4.9 with the latest Security
Update (and as far as I can tell, the extended attributes and ACLs
that I need are being copied properly, so 2007-005 did seem to fix
whatever 004 broke). The script backs up several AFP share points
with a total of about 140GB of data (300,000 files or so). The share
is hosted on an Xserve G5 with 2GB of RAM.
The job ran like a champ for about 3 days (it's set to run every few
hours), and then last night it stopped running properly. It was
exiting with code 22 (Error allocating core memory buffers). I added
the verbose flag to the script this morning and ran it again in the
terminal and about 85% of the way through this started popping up:
rsync(7321) malloc: *** vm_allocate(size=8421376) failed (error code=3)
rsync(7321) malloc: *** error: can't allocate region
rsync(7321) malloc: *** set a breakpoint in szone_error to debug
This continues for about a minute or so (seemingly continuing to
copy files, but that's suspect) until finally the process crashed
and this was spit out:
rsync(7563) malloc: *** vm_allocate(size=8421376) failed (error code=3)
rsync(7563) malloc: *** error: can't allocate region
rsync(7563) malloc: *** set a breakpoint in szone_error to debug
ERROR: out of memory in map_ptr
rsync error: error allocating core memory buffers (code 22) at
/SourceCache/rsync/rsync-24.1/rsync/util.c(120)
rsync: writefd_unbuffered failed to write 105 bytes: phase "unknown"
[generator]: Broken pipe (32)
rsync error: error in rsync protocol data stream (code 12) at
/SourceCache/rsync/rsync-24.1/rsync/io.c(909)
When this happens, rsync has allocated (according to Activity
Monitor) 1.52GB of real memory, 1.89GB of virtual memory and the
total VM is approaching but not quite at 8GB (7.91GB). It also
happens near the same point, i.e. pretty consistently within the
same folder, but with different files (I'm assuming because of
changing memory use during each subsequent run).
So. Is this a VM limitation of the G5? Of the OS? A real memory
limitation? Rsync bugginess that Apple needs to fix? If it is an OS
limitation, is it one that can be removed by editing a config file?
