Re: 10.6.2 issues with F_NOCACHE?
Re: 10.6.2 issues with F_NOCACHE?
- Subject: Re: 10.6.2 issues with F_NOCACHE?
- From: Marc Van Olmen <email@hidden>
- Date: Mon, 8 Feb 2010 14:30:08 -0500
Hi,
I made some quick prototype of the suggestions and I have now valloc buffer.
It seems work well on most systems, but still we feel that the OS is caching on system that have large amount of free memory.
For example we have 2 systems with 9 GB of RAM.
And on those systems it looks like the OS still caches the files, because after a good 20 mins of playing files, we lost 5GB of free memory on those systems.
Until the system has about 1.6GB of free memory left and then it seems to stabilize around that amount.
On a machine with 3 GB of RAM the Free memory stabilize around 1.3GB of RAM.
The memory usage of our app is about 1GBytes.
When I look at the fs_usage output. it looks like both us and quicktime are opening the files with <CACHING OFF> (second one is quicktime F=13)
14:05:02.497 open F=12 (R_____) /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000026 Our App
14:05:02.497 fcntl F=12 <CACHING OFF> 0.000003 Our App
14:05:02.497 write F=6 B=0x96 0.000008 Our App
14:05:02.497 fcntl F=12 <CMD=45> 0.000001 Our App
14:05:02.497 write F=6 B=0xbc 0.000007 Our App
14:05:02.497 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000016 Our App
14:05:02.497 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000021 Our App
14:05:02.497 getattrlist 0.000031 Our App
14:05:02.497 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000021 Our App
14:05:02.497 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000007 Our App
14:05:02.497 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000008 Our App
14:05:02.497 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000008 Our App
14:05:02.497 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000007 Our App
14:05:02.497 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000008 Our App
14:05:02.497 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000007 Our App
14:05:02.497 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000007 Our App
14:05:02.497 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000008 Our App
14:05:02.497 open F=13 (R_____) /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000011 Our App
14:05:02.497 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000008 Our App
14:05:02.498 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000009 Our App
14:05:02.498 getattrlist 0.000006 Our App
14:05:02.498 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000008 Our App
14:05:02.498 fcntl F=13 <CACHING OFF> 0.000002 Our App
14:05:02.503 pread F=13 B=0x8 O=0x00000000 0.005277 W Our App
14:05:02.503 pread F=13 B=0x20 O=0x00000000 0.000304 W Our App
14:05:02.503 pread F=13 B=0x8 O=0x00000020 0.000260 W Our App
14:05:02.504 pread F=13 B=0x8 O=0x00000028 0.000197 W Our App
14:05:02.511 pread F=13 B=0x8 O=0x3b164f40 0.007767 W Our App
14:05:02.512 pread F=13 B=0x8 O=0x3b1680e1 0.000323 W Our App
14:05:02.512 pread F=13 B=0x8 O=0x3b1680e9 0.000202 W Our App
14:05:02.512 pread F=13 B=0x8 O=0x3b1680f5 0.000186 W Our App
14:05:02.512 pread F=13 B=0x8 O=0x3b16b520 0.000251 W Our App
14:05:02.514 getattrlist /video/clips_1080i_PAL/Softron_TestPattern_PRORESHD_i50.mov 0.000032 Our App
14:05:02.515 getattrlist /video/clips_1080i_PAL/Softron_TestPattern_PRORESHD_i50.mov 0.000020 Our App
14:05:02.516 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000012 Our App
14:05:02.516 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000013 Our App
14:05:02.520 pread F=13 B=0x34ca O=0x3b16b528 0.007802 W Our App
14:05:02.521 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000078 Our App
14:05:02.521 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000008 Our App
14:05:02.521 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000031 Our App
14:05:02.522 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000018 Our App
14:05:02.522 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000009 Our App
14:05:02.522 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000007 Our App
14:05:02.522 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000009 Our App
14:05:02.522 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000008 Our App
14:05:02.522 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000032 Our App
14:05:02.522 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000007 Our App
14:05:02.522 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000010 Our App
14:05:02.522 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000007 Our App
14:05:02.522 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000009 Our App
14:05:02.522 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000007 Our App
14:05:02.522 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000009 Our App
14:05:02.522 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000010 Our App
14:05:02.522 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000008 Our App
14:05:02.522 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000007 Our App
14:05:02.522 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000025 Our App
14:05:02.522 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000007 Our App
14:05:02.523 getattrlist /video/clips_1080i_PAL/2500_i_1920x1080_AppleProRes.mov 0.000012 Our App
14:05:02.523 getattrlist 0.000010 Our App
14:05:02.523 getattrlist /video/clips_1080i_PAL 0.000008 Our App
14:05:02.523 getattrlist /video 0.000007 Our App
14:05:02.523 getattrlist /video 0.000006 Our App
14:05:02.523 getattrlist /video 0.000005 Our App
14:05:02.523 getattrlist /video/clips_1080i_PAL 0.000034 Our App
14:05:02.523 getattrlist /video/clips_1080i_PAL 0.000006 Our App
14:05:02.523 getattrlist /video/clips_1080i_PAL 0.000006 Our App
14:05:02.523 getattrlist /video/clips_1080i_PAL 0.000006 Our App
14:05:02.523 close F=13 0.000013 Our App
14:05:02.523 write F=6 B=0x7c 0.000026 Our App
14:05:02.523 write F=6 B=0x9e 0.000006 Our App
14:05:02.523 write F=6 B=0x8e 0.000018 Our App
14:05:02.523 write F=6 B=0x98 0.000006 Our App
14:05:02.523 write F=6 B=0x86 0.000007 Our App
14:05:02.523 write F=6 B=0x78 0.000005 Our App
14:05:02.523 write F=6 B=0x87 0.000006 Our App
14:05:02.524 write F=6 B=0x79 0.000006 Our App
14:05:02.524 write F=6 B=0x85 0.000017 Our App
14:05:02.524 write F=6 B=0x76 0.000006 Our App
14:05:02.537 pread F=12 B=0x2ee00 O=0x00000200 0.012757 W Our App
14:05:02.537 write F=6 B=0x7d 0.000023 Our App
14:05:02.548 pread F=12 B=0x2ee00 O=0x00fd8400 0.011340 W Our App
14:05:02.888 pread F=12 B=0x69aa00 O=0x0008cc00 0.093243 W Our App
14:05:02.895 write F=6 B=0x7e 0.000040 Our App
14:05:02.895 pread F=12 B=0x2ee00 O=0x01f98e00 0.000167 Our App
* The size of read I'm doing is not always a multiple of 4096. I'm wondering if that could play a role?
* Also is it possible to see with fs_usage when something is added to global file cache?
regards,
marc
Quick File Prototype code that implements suggested idea:
//-------------------------------------------------------------------------------------------
// cNoCachedFile::Open
//-------------------------------------------------------------------------------------------
OS_Error cNoCachedFile::Open ( OS_Char * theFilePath )
{
OS_Error anError = kOS_NoError;
int aControlReturn = 0;
fFileRef = open(theFilePath,O_RDONLY);
if (fFileRef == -1) {
anError = kOS_Error_CouldntOpenFile;
goto bail;
}
aControlReturn = fcntl(fFileRef, F_NOCACHE, 1);
if (aControlReturn < 0)
{
#if FW_LOG_PROGRESS
OS_OPTIONAL_DEBUG_LOG(gLog_Progress_On,">>>> cNoCachedFile::Open Open movie: No cache flag gave error %d \n",errno);
#endif
}
aControlReturn = fcntl(fFileRef, F_RDAHEAD, 0);
if (aControlReturn < 0)
{
#if FW_LOG_PROGRESS
OS_OPTIONAL_DEBUG_LOG(gLog_Progress_On,">>>> cNoCachedFile::Open Open movie: No Read a Head flag gave error %d \n",errno);
#endif
}
bail:
return anError;
} // cNoCachedFile::Allocate
//-------------------------------------------------------------------------------------------
// cNoCachedFile::Read
//-------------------------------------------------------------------------------------------
OS_Error cNoCachedFile::Read ( OS_Pointer pStoragePtr,
OS_StreamSize pFilePosition,
OS_MemorySize pDataSize)
{
OS_Error anError = kOS_NoError;
off_t aSeekOffset;
static void* gBufferPtr = NULL;
static OS_MemorySize gBufferPtrSize = 0;
ssize_t aLength;
FW_ASSERT(fFileRef != -1);
FW_ASSERT(gNoCachedFileMutex != NULL);
// We can only read one block at the time, we should do this better and actually use the timestamp.
// And then schedule which read goes first by timestamp. now we rely on the Kernel Thread scheduler.
gNoCachedFileMutex->Lock();
if ( (gBufferPtr == NULL)
|| (gBufferPtrSize < pDataSize))
{
if (gBufferPtr)
{
free(gBufferPtr);
}
else if (gBufferPtrSize == 0)
{
gBufferPtrSize = 8 * 1024 * 4096;
}
if (gBufferPtrSize < pDataSize)
{
gBufferPtrSize = pDataSize % 4096;
if (gBufferPtrSize > 0)
{
gBufferPtrSize = pDataSize - pDataSize % 4096 + 4096;
}
else
{
gBufferPtrSize = pDataSize;
}
}
gBufferPtr = valloc(gBufferPtrSize);
FW_ASSERT(gBufferPtr != NULL);
#if FW_LOG_PROGRESS
OS_OPTIONAL_DEBUG_LOG(gLog_Progress_On,">>>> cNoCachedFile::Read allocated %u read buffer: %d \n",gBufferPtr,gBufferPtrSize);
#endif
}
aLength = pread(fFileRef, gBufferPtr, pDataSize,pFilePosition);
memcpy(pStoragePtr, gBufferPtr, pDataSize);
gNoCachedFileMutex->Unlock();
if (aLength < 0)
{
anError = errno;
if (anError == kOS_NoError)
{
anError = kOS_MovieError_FileReadError;
}
}
return anError;
}
//-------------------------------------------------------------------------------------------
// cNoCachedFile::Close
//-------------------------------------------------------------------------------------------
void cNoCachedFile::Close ( )
{
if (fFileRef != -1)
{
close(fFileRef);
fFileRef = -1;
}
}
On Feb 4, 2010, at 12:10 PM, email@hidden wrote:
> Chris,
>
> Thanks for the input, I will do some test tonight.
>
> I know in my case this isn't, that's why I mentioned this:
>
> --------------
> Not sure important: my memory pointer that i pass into the read, is an
> offset in a section of a big memory blob allocated with malloc.
> --------------
>
> Question: is it sufficient that the pointer address I pass is a multiple
> of 4096 (in case page size on that platform is 4096) or do I have allocate
> the memory in a special way. Because as far I recall Malloc creates and
> aligned buffer, but not a page aligned buffer.
>
> thanks,
>
> marc
>
>
>
>> Marc --
>>
>> To ensure proper F_NOCACHE behavior, you should really ensure you pass
>> page-aligned buffers to read/pread. The buffer cache may decide it has
>> to go through the copy path if your buffer is not well-aligned and
>> your DMA controller requires better alignment.
>>
>> -- Chris
>>
>>
>> On Feb 3, 2010, at 8:53 PM, Marc Van Olmen wrote:
>>
>>> hi,
>>>
>>> Just trying to trace an issue with my app:
>>>
>>> Our app runs fine under 10.5.8 (just retested this)
>>>
>>> The app plays a playlist of HD videos and about 20Mbytes/second data
>>> rate. It plays a few of those clips in loop.
>>> I'm using the standard unix open function to read from these files.
>>>
>>> fFileRef = open(fFilePath,O_RDONLY);
>>>
>>> if (fFileRef == -1) {
>>> anError = kOS_Error_CouldntOpenFile;
>>> goto bail;
>>> }
>>> BAIL_OSERROR(fcntl(fFileRef, F_NOCACHE, 1));
>>>
>>> (Also QuickTime has these files open, but I don't use Quicktime to
>>> read) but maybe this is a cause.
>>>
>>> There is only one file open at the time. Also my app doesn't
>>> allocate much memory once the video is playing. So the Application
>>> memory is constant for most of the time.
>>>
>>>
>>>
>>> Under 10.6.2 I see something strange happening.Under Activity
>>> Memory: the System Memory->Free is decreasing after playing above
>>> clips for like 20 min.
>>> Under System 10.5.8 this stays constant.
>>>
>>> My system has 3GB of ram. Under 10.5.8 this stays around 1.5GB free
>>> available. Under 10.6.2 this goes to 200Mbytes after 20 min.
>>>
>>> After a while I notice the "Swap used" starts to grow a little under
>>> 10.6.2 and this causes sometimes frame drops in my video playback
>>> app. Under 10.5.8 this doesn't happen.
>>>
>>>
>>> When I go to the Terminal (10.6.2) and I use the 'purge' command
>>> Activity Memory: the System Memory->Free goes back 1.2 GBytes Free
>>> memory.
>>>
>>> So this makes me assume that disk cache is being filled by my reading.
>>>
>>> I also tested this on a system with 6GB and above symptoms happening
>>> also.
>>>
>>> I also tried doing this:
>>>
>>> BAIL_OSERROR(fcntl(fFileRef, F_RDAHEAD, 0));
>>>
>>> no difference. Same behavior under 10.6.2
>>>
>>> I tried:
>>> BAIL_OSERROR(fcntl(fFileRef, F_GLOBAL_NOCACHE, 1));
>>>
>>> no difference. Same behavior under 10.6.2
>>>
>>> Not sure important: my memory pointer that i pass into the read, is
>>> an offset in a section of a big memory blob allocated with malloc.
>>>
>>>
>>> * Any suggestion on how to get rid of this problem? I
>>> * Did something changed under 10.6 regarding F_NOCACHE?
>>> * Any suggestion on how trace this issue better? Is there a way I
>>> can trace when a read data is being added to the disk cache for
>>> example?
>>> I didn't see anything in Instruments->Disk Activity. Maybe I missed
>>> something.
>>>
>>> I can file a radar in case needed but just double checking if I'm
>>> overlooking something simple here.
>>>
>>> cheers,
>>>
>>> marc
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Do not post admin requests to the list. They will be ignored.
>>> Darwin-dev mailing list (email@hidden)
>>> Help/Unsubscribe/Update your Subscription:
>>>
>>> This email sent to email@hidden
>>
>>
>> ------------------
>> 6 Infinite Loop
>> M/S 306-2MS
>> Cupertino CA 95014
>> phone: (408) 974-4033
>> fax: (408) 862-7577
>> email: email@hidden
>>
>>
>
>
------------------
6 Infinite Loop
M/S 306-2MS
Cupertino CA 95014
phone: (408) 974-4033
fax: (408) 862-7577
email: email@hidden
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Darwin-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden