Hi Charlie,
On Feb 3, 2010, at 1:08 PM, charlie strauss wrote:
> Hello,
>
> xgrid is crashing on me when the input directory is too large. Specifically if I try to execute:
>
> xgrid -in foo -submit hello.sh
>
> where foo is a directory that will be passed to the agent.
>
> if foo contains a file of size 395MB then all this works.
> if foo contains two or more files of size 395MB then the xgrid command above crashes with an NSException from a malloc error.
>
> My understanding is that all I/O needs to be less than 2GB. but it seems like 800MB is indeed less than 2MB so I'm puzzled.
Xgrid is not designed for transferring large binary files. If I recall correctly, it uses Base64 encoding which (along with the metadata) could be the source of your problems.
You really want a shared filesystem and Kerberos authentication; Xgrid was never meant to be a substitute for either of those.
-- Ernie P.
>
> I'm new to this so I'm trying to understand what my limitations are.
>
> here is a reproducible sample case where it crashes with three such large files. it is crashing in the client's submit not in the controller or the agent run time. (In this case the client, agent and the controller are the same computer with 4GB.)
>
> ls -l
> -rw-r--r-- 1 rooms staff 413937664 Feb 3 12:01 bigfella
> -rw-r--r-- 1 rooms staff 413937664 Feb 3 12:02 bigfella0
> -rw-r--r-- 1 rooms staff 413937664 Feb 3 12:02 bigfella1
> -rwxr-xr-x@ 1 rooms staff 197 Feb 3 11:58 hello.sh
>
>
> xgrid:grid rooms$ xgrid -in /Users/rooms/grid -job submit hello.sh
>
> xgrid(38806,0x7fff70e94be0) malloc: *** mmap(size=18446744071562067968) failed (error code=12)
> *** error: can't allocate region
> *** set a breakpoint in malloc_error_break to debug
> 2010-02-03 12:08:55.967 xgrid[38806:903] *** Terminating app due to uncaught exception 'NSMallocException', reason: 'Attempt to allocate -2147483648 bytes for NS/CFData failed. Maximum size: 4398046511103'
> *** Call stack at first throw:
> (
> 0 CoreFoundation 0x00007fff831ec5a4 __exceptionPreprocess + 180
> 1 libobjc.A.dylib 0x00007fff86a9d313 objc_exception_throw + 45
> 2 Foundation 0x00007fff83b6a1d2 _NSSearchForNameInPath + 0
> 3 CoreFoundation 0x00007fff83216396 __CFDataHandleOutOfMemory + 86
> 4 CoreFoundation 0x00007fff83176139 __CFDataGrow + 713
> 5 CoreFoundation 0x00007fff8317519a CFDataReplaceBytes + 378
> 6 CoreFoundation 0x00007fff831c73d7 _CFAppendXML0 + 1751
> 7 CoreFoundation 0x00007fff831c7055 _CFAppendXML0 + 853
> 8 CoreFoundation 0x00007fff831c7055 _CFAppendXML0 + 853
> 9 CoreFoundation 0x00007fff831c7055 _CFAppendXML0 + 853
> 10 CoreFoundation 0x00007fff831c7055 _CFAppendXML0 + 853
> 11 CoreFoundation 0x00007fff831c7055 _CFAppendXML0 + 853
> 12 CoreFoundation 0x00007fff831c6af3 _CFPropertyListCreateXMLData + 147
> 13 CoreFoundation 0x00007fff8316832d CFPropertyListWrite + 285
> 14 CoreFoundation 0x00007fff83167854 CFPropertyListWriteToStream + 244
> 15 Foundation 0x00007fff83a8d50d +[NSPropertyListSerialization dataFromPropertyList:format:errorDescription:] + 79
> 16 xgrid 0x000000010000908a 0x0 + 4295004298
> 17 xgrid 0x0000000100008f64 0x0 + 4295004004
> 18 Foundation 0x00007fff83a60926 _nsnote_callback + 167
> 19 CoreFoundation 0x00007fff83194e4a __CFXNotificationPost + 954
> 20 CoreFoundation 0x00007fff831817f8 _CFXNotificationPostNotification + 200
> 21 Foundation 0x00007fff83a78500 __NSFireDelayedPerform + 404
> 22 CoreFoundation 0x00007fff83189a78 __CFRunLoopRun + 5480
> 23 CoreFoundation 0x00007fff8318803f CFRunLoopRunSpecific + 575
> 24 Foundation 0x00007fff83a9ba94 -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 270
> 25 Foundation 0x00007fff83a9b973 -[NSRunLoop(NSRunLoop) run] + 77
> 26 xgrid 0x0000000100001121 0x0 + 4294971681
> 27 xgrid 0x0000000100001040 0x0 + 4294971456
> 28 ??? 0x000000000000000c 0x0 + 12
> )
> terminate called after throwing an instance of 'NSException'
> Abort trap
>
>
>
>
>
> _______________________________________________
> Do not post admin requests to the list. They will be ignored.
> Xgrid-users mailing list (email@hidden)
> Help/Unsubscribe/Update your Subscription:
>
> This email sent to email@hidden
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Xgrid-users mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden
