Re: Identify image file count in directory
Re: Identify image file count in directory
- Subject: Re: Identify image file count in directory
- From: "Gary L. Wade" <email@hidden>
- Date: Sat, 14 Nov 2015 10:56:38 -0800
I've worked on backup, syncing apps, and other file system related utilities for some rather well-known companies, so I've definitely explored the speed issues you're seeing. At one time I also had explored and used the CoreOS APIs that were shared with the legacy Carbon system (FSIterator, etc.). One app I refactored to not use Cocoa for just getting the listing of a large file system dropped from three minutes to 20 seconds and significantly reduced the memory footprint in the process.
The Cocoa frameworks pretty much call these lower level APIs and encapsulate their results in Cocoa objects (i.e., HFS+ is not composed of Cocoa objects), which can indeed add up when you're looking at the quantity of files you're seeing, and uses a lot more memory than needed for your case, which is just counting the files that are images, essentially checking the extensions from a known set (getting an extension from an NSString is also more expensive than traversing a C array of bytes to find the last period). And when you're going across a network boundary, the speed in doing more than needed really adds up.
--
Gary L. Wade (Sent from my iPad)
http://www.garywade.com/
> On Nov 14, 2015, at 10:15 AM, Jonathan Taylor <email@hidden> wrote:
>
> Hi Gary,
>
> Thanks for your reply. Do you know for sure that this should be significantly faster, or were you just guessing? You didn't say which particular API you had in mind, but some reading has led me to the fts APIs. These appear to be about 20% faster than NSFileManager (I specified FTS_NAMEONLY). Better than a kick in the teeth, but still seems to me like it might be possible to do it faster.
>
> I measured about 2.5 seconds for a directory on a "Mac OS Extended (Journaled)" external USB3 hard disk containing 110,000 files with names around 15 characters long. I don't know much about how filesystems are structured, so this information may not be all in one place, but that should represent less than 1MB/s of data read in order to get a list of all the filenames in the directory.
>
> Maybe this is as good as it's going to get, but it would be great to know if there is a better way.
>
> Jonny.
>
>> On 13 Nov 2015, at 16:54, Gary L. Wade <email@hidden> wrote:
>>
>> Try going down a level to the BSD layer APIs for directory contents traversal.
>> --
>> Gary L. Wade (Sent from my iPad)
>> http://www.garywade.com/
>>
>>> On Nov 13, 2015, at 8:28 AM, Jonathan Taylor <email@hidden> wrote:
>>>
>>> Hi all,
>>>
>>> I want to be able to identify quickly (programatically) how many image files reside in a particular directory. At present I call:
>>> [NSFileManager defaultManager] contentsOfDirectoryAtPath:dir error:nil];
>>> and then examine the type suffixes (which in comparison is very quick). When looking at a directory over a network or on an external drive, the NSFileManager call can take several seconds for a directory containing 18k files of which half are images.
>>>
>>> These sorts of numbers are in fact a common use case for me, and I would like to avoid this delay. This is for preview information in an NSOpenPanel, so I don't want to make things this unresponsive - but at the same time it is very useful to have access to this information for the preview.
>>>
>>> Can anybody advise on a quicker way of achieving what I want to achieve? The fact that 'ls' takes almost as long makes me think this is probably a fairly insurmountable problem, but at the same time the quantity of information transferred (of the order of 200k of data) should not take 2 seconds to transfer, so in that sense it doesn't seem unreasonable to try and see if there is a faster way.
>>>
>>> I would prefer to get the filenames themselves, but I could settle for just a count of the total number of files (of any kind) in the directory *and* the ability to get the paths of just the first few files in the directory, if there might be a faster way of doing that.
>>>
>>> Thanks for any suggestions
>>> Jonny
>
_______________________________________________
Cocoa-dev mailing list (email@hidden)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden