User-agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7.2) Gecko/20040804 Netscape/7.2
On 7/06/07 22:20, Douglas Back wrote:
I have a launchd job that runs a little shell bash script for some
rsync backup action. I'm running 10.4.9 with the latest Security
Update (and as far as I can tell, the extended attributes and ACLs
that I need are being copied properly, so 2007-005 did seem to fix
whatever 004 broke). The script backs up several AFP share points
with a total of about 140GB of data (300,000 files or so). The share
is hosted on an Xserve G5 with 2GB of RAM.
300,000 files should not be that terrible for rsync.
But, as shipped by Apple, it leaks; and it leaks much, much more when it
encounters items with ACLs (just noticed the fact recently).
Do your file/folders all have an ACL?
The job ran like a champ for about 3 days (it's set to run every few
hours), and then last night it stopped running properly.
Has the file count increased during those 3 days?
(this was in fact another way to ask he usual question: has something
changed before/after? :-) )
It was exiting with code 22 (Error allocating core memory buffers). I
added the verbose flag to the script this morning and ran it again in
the terminal and about 85% of the way through this started popping up:
rsync(7321) malloc: *** vm_allocate(size=8421376) failed (error code=3)
rsync(7321) malloc: *** error: can't allocate region
rsync(7321) malloc: *** set a breakpoint in szone_error to debug
Yep. Because of the previous leaks, rsync became unable to build its
memory structures for preparing its AppleDouble files.
[...]
On 17/06/07 4:59, Jackson, Phillip ((Phil)) wrote:
We have a recently implemented Apple Xsan which we plan to backup to
NAS.
We would like to use rsync to perform incremental, scheduled
backups. The caveat, however, is that we currently have over 5
million + individual files that require backing up. We're running
into the following error when running rsync:
[...]
Assuming rsync would run, the amount of time required to reference
both the source/destination volume is prohibitive, taking several hours.
Yes, the more leaks, the more prohibitive it becomes to allocate another
chunk of memory...
Does anyone have any suggestions as to how we should approach backing
up this volume from the command line?
If you are satisfied with rsync's capabilities as of 10.4.9, I have
today amended my rsync binary (see my post dated May, 6th) so as to
avoid -or at least, minimize- memory leaks.
I haven't tested it against 5 million files, but my own trials against
1,5 million files with lots of them having an ACL tend to show it should
work with much more files.
I would be a pleasure to send a copy of the binary to anyone interested
in trying it.
BTW, in the case of an Xsan volume, what arguments are you using with rsync?
HTH,
Axel
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Macos-x-server mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/macos-x-server/email@hidden
