So. Is this a VM limitation of the G5? Of the OS? A real memory
limitation? Rsync bugginess that Apple needs to fix? If it is an OS
limitation, is it one that can be removed by editing a config file?
AFAICT, this is a limitation in rsync. According to their FAQ:
rsync creates a list of all the files to transfer before it starts.
Each file entry takes 100 bytes, so if you have a lot of files, the
file list can become so large that it overwhelms the amount of memory
you have. Note that the amount of data is irrelevant; it's the
number of discrete files that matters.
Now, I'm not sure if you're seeing the exact same thing, because you
claim to only have 300,000 files. To chew up 1.5GB of RAM, you'd
theoretically need 16 million files. However, Apple's rsync may
store more info in the file list due to the ACLs, so perhaps the
memory exhaustion happens faster.
Are you sure you only have 300,000 files? Also, can you confirm when
the crash is happening (during the list building phase, or during the
actual copy phase)? Try adding the --stats arg to rsync to see what
your file list sizes are like.
When we encountered this problem (10 million+ files), we had to re-
write our backup script to iterate over a list of directories and
make a separate rsync call for each one. That broke up the file
lists enough to keep the memory usage under control.
