Chris,
I was able to isolate the scenario of the bug: This piece of code works fine on system 10.4 and 10.5.8 but under snowleopard this code her uses quickly all the Free Memory on the mac.
Important the file that I picked is a file of about 8 GB in size. I notice that this doesn't happen when the file is smaller then 4 GB.
I have done days of testing and It is always reproducible on the 5 machines I used for testing all those machines are running 10.6.2
Machines I uses were MacPro's First intel model to last Intel Model and Macbook Pro's 15 inch.
If this code runs for about 2 mins on My MacPro 3GB of RAM, suddenly in less the 2 seconds my Free Memory will go down from 1.7 GB to 100MBytes.
On my MacBook Pro 4GBytes. RAM it will take about 2 min and 30 seconds and then it free Memory of 2.5GB will go down to 100MBytes.
Om other Machine with 10GBytes of RAM it took about 10 Min 34 seconds always around the same time the symptoms happen.
This is very bad in real time video playback because such a low amount of free memory can give frame drops as soon anything else happens on the mac.
So this bug is very important to be fixed.
I will file a radar bug tomorrow
After days of testing I came to following criterea
* file must be more then 4Gbytes
* You have to read at high data rates like 40 MBytes/Second. I tried with 20MBytes per second.
* You have to walk through file with weird offsets this produces it the most quickest. I tried 40*1021*1024 + 21 seems to produce it quick.
The code is actually super simple.
#import <Cocoa/Cocoa.h>
@interface TestFileReadAppDelegate : NSObject {
NSWindow * window;
NSTimer * mReadTimer;
long long mStartOffset;
int mFileRef;
}
@property (assign) IBOutlet NSWindow *window;
@end
#define kInitialeVAllocBufferSize 80 * 1024 * 1024
#define kReadSize 40 * 1024 * 1024 // You can also do '40 * 1024 * 1024 + 21'
#define kNextReadSizeOffset 21 // and have this value 0
@implementation TestFileReadAppDelegate
@synthesize window;
-(void)_readFromFile:(id)theSender
{
if (mFileRef > 0)
{
size_t aLength;
static void* gBufferPtr = NULL;
if (gBufferPtr == NULL)
{
gBufferPtr = valloc(kInitialeVAllocBufferSize);
}
aLength = pread(mFileRef, gBufferPtr, kReadSize,mStartOffset);
mStartOffset += kReadSize + kNextReadSizeOffset;
if (aLength < 0)
{
// End of file.
close(mFileRef);
[mReadTimer invalidate];
}
}
}
-(void)open: (char *) theFilePath
{
int aControlReturn = 0;
mFileRef = open(theFilePath,O_RDONLY);
mStartOffset = 0;
if (mFileRef > -1)
{
aControlReturn = fcntl(mFileRef, F_NOCACHE, 1);
}
else
{
}
}
- (void)applicationDidFinishLaunching:(NSNotification *)aNotification
{
// Insert code here to initialize your application
[self open:"/ProRes8min.mov"];
mReadTimer = [[NSTimer timerWithTimeInterval:1.0 target: self selector:@selector(_readFromFile:) userInfo:nil repeats:YES] retain];
[[NSRunLoop currentRunLoop] addTimer:mReadTimer forMode: NSDefaultRunLoopMode];
}
@end
cheers,
marc
On Feb 8, 2010, at 3:05 PM, Marc Van Olmen wrote:
Chris,
Thanks for your feedback.
Marc --
A small snippet of fs_usage trace is not enough to conclude what entity is caching data. However, given our knowledge of ProApps workloads and how the system behaves when using apps like Final Cut Pro, I think the burden of proof lies on the developer to isolate their own code and see if it's causing the issue.
I understand but my questions is how does fs_usage reports a Cache hit in file cache log? I couldn't read that information in the documentation.
Within this code you seem to be allocating memory and freeing it, and copying data which was directly DMA'd into it to application supplied buffers. Both of those are not going to be very efficient for any sort of rich media application.
The memory is allocated once, during the application. The buffer is declared static, so this buffer stays alive on the next read. Like I said it just a quick prototype to see if a valloc buffer would solve the issue.
You really want to have a preallocated pool of buffers for use so you aren't constantly calling valloc()/free(). In addition, I'm not sure from the code snippet provided why you can't just read directly into the buffer passed into the Read() method. That would be the most efficient way to do a read with minimal copying. That is the point of direct I/O, right?
Currently the buffer passed in the read function is not valloc allocated. It is part of big Memory Buffer and the read buffer is an offset in that currently it is not page-aligned. The engine uses its own memory allocator optimized for video playback doesn't do malloc/free during playback. Code is written to work well on a G4. So is highly optimized for those old days.
I understand I'm not doing the smartest thing, I just made a quick prototype to see if I the valloc technique gave better result. So I use the global valloc buffer to read and then copy it into the buffer passed on by read.
In the future will solve this in the proper way and make sure my buffer passed in the Read functions is page-aligned.
Finally, I believe the fcntl for F_RDAHEAD is redundant. I believe F_NOCACHE implies the system will only issue physical I/Os for the supplied buffer.
Good to know,
Thanks
marc
Thanks,
-- Chris
------------------
6 Infinite Loop
M/S 306-2MS
Cupertino CA 95014
phone: (408) 974-4033
fax: (408) 862-7577