February 2010 - Darwin-dev - lists.apple.com

Re: 10.6.2 issues with F_NOCACHE?
by Michael Smith 05 Feb '10

05 Feb '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: 10.6.2 issues with F_NOCACHE?
by Chris Sarcone 04 Feb '10

04 Feb '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: 10.6.2 issues with F_NOCACHE?
by mvo＠sky4studios.be 04 Feb '10

04 Feb '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Importance: Normal User-agent: SquirrelMail/1.4.15 Chris, Thanks for the input, I will do some test tonight. I know in my case this isn't, that's why I mentioned this: -------------- Not sure important: my memory pointer that i pass into the read, is an offset in a section of a big memory blob allocated with malloc. -------------- Question: is it sufficient that the pointer address I pass is a multiple of 4096 (in case page size on that platform is 4096) or do I have allocate the memory in a special way. Because as far I recall Malloc creates and aligned buffer, but not a page aligned buffer. thanks, marc > Marc -- > > To ensure proper F_NOCACHE behavior, you should really ensure you pass > page-aligned buffers to read/pread. The buffer cache may decide it has > to go through the copy path if your buffer is not well-aligned and > your DMA controller requires better alignment. > > -- Chris > > > On Feb 3, 2010, at 8:53 PM, Marc Van Olmen wrote: > >> hi, >> >> Just trying to trace an issue with my app: >> >> Our app runs fine under 10.5.8 (just retested this) >> >> The app plays a playlist of HD videos and about 20Mbytes/second data >> rate. It plays a few of those clips in loop. >> I'm using the standard unix open function to read from these files. >> >> fFileRef = open(fFilePath,O_RDONLY); >> >> if (fFileRef == -1) { >> anError = kOS_Error_CouldntOpenFile; >> goto bail; >> } >> BAIL_OSERROR(fcntl(fFileRef, F_NOCACHE, 1)); >> >> (Also QuickTime has these files open, but I don't use Quicktime to >> read) but maybe this is a cause. >> >> There is only one file open at the time. Also my app doesn't >> allocate much memory once the video is playing. So the Application >> memory is constant for most of the time. >> >> >> >> Under 10.6.2 I see something strange happening.Under Activity >> Memory: the System Memory->Free is decreasing after playing above >> clips for like 20 min. >> Under System 10.5.8 this stays constant. >> >> My system has 3GB of ram. Under 10.5.8 this stays around 1.5GB free >> available. Under 10.6.2 this goes to 200Mbytes after 20 min. >> >> After a while I notice the "Swap used" starts to grow a little under >> 10.6.2 and this causes sometimes frame drops in my video playback >> app. Under 10.5.8 this doesn't happen. >> >> >> When I go to the Terminal (10.6.2) and I use the 'purge' command >> Activity Memory: the System Memory->Free goes back 1.2 GBytes Free >> memory. >> >> So this makes me assume that disk cache is being filled by my reading. >> >> I also tested this on a system with 6GB and above symptoms happening >> also. >> >> I also tried doing this: >> >> BAIL_OSERROR(fcntl(fFileRef, F_RDAHEAD, 0)); >> >> no difference. Same behavior under 10.6.2 >> >> I tried: >> BAIL_OSERROR(fcntl(fFileRef, F_GLOBAL_NOCACHE, 1)); >> >> no difference. Same behavior under 10.6.2 >> >> Not sure important: my memory pointer that i pass into the read, is >> an offset in a section of a big memory blob allocated with malloc. >> >> >> * Any suggestion on how to get rid of this problem? I >> * Did something changed under 10.6 regarding F_NOCACHE? >> * Any suggestion on how trace this issue better? Is there a way I >> can trace when a read data is being added to the disk cache for >> example? >> I didn't see anything in Instruments->Disk Activity. Maybe I missed >> something. >> >> I can file a radar in case needed but just double checking if I'm >> overlooking something simple here. >> >> cheers, >> >> marc >> >> >> >> >> >> >> _______________________________________________ >> Do not post admin requests to the list. They will be ignored. >> Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) >> Help/Unsubscribe/Update your Subscription: >> http://lists.apple.com/mailman/options/darwin-dev/sarcone%40apple.com >> >> This email sent to sarcone(a)apple.com > > > ------------------ > 6 Infinite Loop > M/S 306-2MS > Cupertino CA 95014 > phone: (408) 974-4033 > fax: (408) 862-7577 > email: sarcone(a)apple.com > > _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: 10.6.2 issues with F_NOCACHE?
by Chris Sarcone 04 Feb '10

04 Feb '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: 10.6.2 issues with F_NOCACHE?
by Jonas Maebe 04 Feb '10

04 Feb '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On 04 Feb 2010, at 17:41, Jens Alfke wrote: > Couldn't one mmap() only a piece of the file at a time? I.e. mmap the first 10MB, then unmap it and map the next 10MB, etc. The buffer cache would probably keep the memory that you munmap still around (if you touched it prior to munmapping). I think that's what Godfrey meant with "it will map the entire file into memory at some stage". Jonas _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: 10.6.2 issues with F_NOCACHE?
by Jens Alfke 04 Feb '10

04 Feb '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Feb 4, 2010, at 2:23 AM, Godfrey van der Linden wrote: > Unfortunately the mmap won't really help you as it will map the entire file into memory at some stage. Yes, but more efficiently than reading the entire file into a malloc block (which is what I inferred the OP is doing; but maybe I'm wrong.) Couldn't one mmap() only a piece of the file at a time? I.e. mmap the first 10MB, then unmap it and map the next 10MB, etc. —Jens _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: 10.6.2 issues with F_NOCACHE?
by Godfrey van der Linden 04 Feb '10

04 Feb '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Unfortunately the mmap won't really help you as it will map the entire file into memory at some stage. Have you written a test case, just a little program that walks the files multiple times both with F_NOCACHE and without it? top will give you a feel for how the memory system is progressing. Cheers Godfrey On 2010-02-04, at 5:16 PM, Jens Alfke wrote: > > On Feb 3, 2010, at 8:53 PM, Marc Van Olmen wrote: > >> Not sure important: my memory pointer that i pass into the read, is an offset in a section of a big memory blob allocated with malloc. > > Have you considered memory-mapping the file instead? That way you won't be allocating VM backing store for the copy of the file in the heap. > >> * Any suggestion on how trace this issue better? Is there a way I can trace when a read data is being added to the disk cache for example? > > I usually use the fs_usage tool for things like this. (You can probably get similar data from Instruments, but I'm an old fogey and I've never really been able to figure out how to get Instruments to tell me anything useful...) _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: 10.6.2 issues with F_NOCACHE?
by Jens Alfke 04 Feb '10

04 Feb '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Feb 3, 2010, at 8:53 PM, Marc Van Olmen wrote: > Not sure important: my memory pointer that i pass into the read, is an offset in a section of a big memory blob allocated with malloc. Have you considered memory-mapping the file instead? That way you won't be allocating VM backing store for the copy of the file in the heap. > * Any suggestion on how trace this issue better? Is there a way I can trace when a read data is being added to the disk cache for example? I usually use the fs_usage tool for things like this. (You can probably get similar data from Instruments, but I'm an old fogey and I've never really been able to figure out how to get Instruments to tell me anything useful...) —Jens _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

10.6.2 issues with F_NOCACHE?
by Marc Van Olmen 04 Feb '10

04 Feb '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com hi, Just trying to trace an issue with my app: Our app runs fine under 10.5.8 (just retested this) The app plays a playlist of HD videos and about 20Mbytes/second data rate. It plays a few of those clips in loop. I'm using the standard unix open function to read from these files. fFileRef = open(fFilePath,O_RDONLY); if (fFileRef == -1) { anError = kOS_Error_CouldntOpenFile; goto bail; } BAIL_OSERROR(fcntl(fFileRef, F_NOCACHE, 1)); (Also QuickTime has these files open, but I don't use Quicktime to read) but maybe this is a cause. There is only one file open at the time. Also my app doesn't allocate much memory once the video is playing. So the Application memory is constant for most of the time. Under 10.6.2 I see something strange happening.Under Activity Memory: the System Memory->Free is decreasing after playing above clips for like 20 min. Under System 10.5.8 this stays constant. My system has 3GB of ram. Under 10.5.8 this stays around 1.5GB free available. Under 10.6.2 this goes to 200Mbytes after 20 min. After a while I notice the "Swap used" starts to grow a little under 10.6.2 and this causes sometimes frame drops in my video playback app. Under 10.5.8 this doesn't happen. When I go to the Terminal (10.6.2) and I use the 'purge' command Activity Memory: the System Memory->Free goes back 1.2 GBytes Free memory. So this makes me assume that disk cache is being filled by my reading. I also tested this on a system with 6GB and above symptoms happening also. I also tried doing this: BAIL_OSERROR(fcntl(fFileRef, F_RDAHEAD, 0)); no difference. Same behavior under 10.6.2 I tried: BAIL_OSERROR(fcntl(fFileRef, F_GLOBAL_NOCACHE, 1)); no difference. Same behavior under 10.6.2 Not sure important: my memory pointer that i pass into the read, is an offset in a section of a big memory blob allocated with malloc. * Any suggestion on how to get rid of this problem? I * Did something changed under 10.6 regarding F_NOCACHE? * Any suggestion on how trace this issue better? Is there a way I can trace when a read data is being added to the disk cache for example? I didn't see anything in Instruments->Disk Activity. Maybe I missed something. I can file a radar in case needed but just double checking if I'm overlooking something simple here. cheers, marc _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

particulars of catching EXC_CRASH
by Neal Sidhwaney 03 Feb '10

03 Feb '10

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=ORFNzr6QjwFllr+WiaXZWpadTcKVlN5+NRgGnOAWwXg=; b=k2JnhO7I6iVX06TAwlOEuhDFMrbVGQDb1kRE8CPgeV2DFiuSgLMRGt6vckTim2aa7t MAWdDFq+E9/fermB6+Orl1cEP5yTmS6W+QDpxUpvQo/u4B9I7ww/OxBCIUD3NKOZ35p+ uZ41SL3f3zEr0JmhqO0SJAcnnshix+MqLdpHQ= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=maCqEw2bZaALt5oSDA/W4rpcUEMfr8tXCeO4DbgdUSlZVAYrd3YT84KeaYkNqlz/Xi vBDdvrg4IFoCwsp8+J3k6Usp7TuafVQ1hiiISv4jezuHnivI5VTOScbvzENfytiTQs/i suqi+OBjY29j4LbrVi3cKTJbeAUZ+TSIMRmd0= _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0