Re: Slow directory browsing in OS X10.11, was fast on 10.6.
Re: Slow directory browsing in OS X10.11, was fast on 10.6.
- Subject: Re: Slow directory browsing in OS X10.11, was fast on 10.6.
- From: Jim Luther <email@hidden>
- Date: Fri, 03 Jun 2016 11:09:14 -0700
Hi Thomas,
10.6 or 10.7 to 10.11 is a big jump with lots of changes. If you could hone it down to which major release the performance regression occurred (it might have been in 10.8, 10.9, or 10.10), it would help.
Also, an Instruments Time Profiler trace including kernel callstacks should show where the aditional time is being spent.
> On Jun 3, 2016, at 3:50 AM, Thomas Tempelmann <email@hidden> wrote:
>
> As the author of a file search application, I've been contacted by a user who claims that searching the same SMB-mounted volume on 10.6.8 was noticable faster than with 10.11. Furthermore, a second search was significantly faster on 10.6, whereas there was almost no speed-up on 10.11.
>
> For instance, the old system was able to look at all dir entries in 33 seconds while the new system takes over 4 minutes. This is bad.
>
> My app simply recurses thru all folder contents, starting at the volume's root, using FSGetCatalogInfoBulk, fetching only the file names
>
> I understand that...
>
> 1. Using FSRef based APIs requires the FS layer to cache the refs along with their paths (because, at the lowest level, all is still path-based, right?)
FSRefs are not and have never been cached. However, the data used to create an FSRef is cached by either the file system/kernel (for file systems that support VOL_CAP_FMT_PATH_FROM_ID and 32-bit IDs), or by coreservicesd.
And yes, the file system is path-based.
> 2. Since FSRefs have no reference counter, they are cached in an arbitrary way and can get kicked out at any time when the OS decides it doesn't need them any more. This will cause a cache miss next time the same Ref is accessed, leading to a new network request, slowing everything down significantly.
Again, the caching is done at lower layers.
> 3. Since my app needs to access every file only once, there's no need for explicit caching, so there's little reason for me so far to switch to using CFURLs - that's only needed when I have to reference the item later again. And so I use CFURLs only if I have found a match, because then I'll keep referencing that item for displaying it to the user. But for items that do not match the user's search criteria, I don't need to keep them and can let the FS cache drop them as needed.
If you don't need the "track a location by file ID" behavior that FSRefs (and File Reference URLs) provides, path-based API is usually faster. Getting attributes adds to the expense. For example, on my system, if I use CFURLEnumerator to recursively enumerate "/Applications" getting file path URLs, it takes:
320616 urls and 0 errors in 1.780873 seconds
With file reference URLs, it takes:
320616 urls and 0 errors in 2.551695 seconds
And those times don't show the time it would take to convert the file reference URL back to its current POSIX path when it is used.
> But why is 10.11 so much slower now than 10.6?
>
> The 10.6 runs suggest that all the FSRefs stayed in the cache so that the second search was much faster. But with 10.11 this is not the cache any more, so what has changed about the Cache, and can that be controlled to make it work better again?
>
> Mind you, it makes no sense for me app to keep a CFURL for every item that the app searched (i.e. for every item on the volume) because they'd all be freed when the app quits, and thus would not keep them in cache if the user quits the search app and re-runs it a few minutes later. Yet, with 10.6, my app could be quit and still the next search would be faster, i.e. the FSRefs were still cached, apparently.
>
> Also, why has even the initial search become slower on 10.11 vs 10.6? It's the very same program of mine on both systems, with the same attributes requested, so I don't think it's my app that's causing this. Rather something changed about FSRefs caching or general smb handling that has degraded this straight-forward performance.
>
> (Note: The user tried SMB1 on both OS versions, by mounting using cifs: which I believe is the way to force SMB1. But even with SMB2 it still is slower on 10.11 than it was on 10.6, he says).
>
> The server is a Windows Server, BTW.
>
> Actually, anohter customer just contacted me who noticed the same slowdown at even greater intensity when he upgraded from 10.7 (I believe) to 10.11. His disk is near-full, so probably has a highly fragmented directory, and he has only 8 GB of RAM (while I have 20). Acitivity monitor shows that the "Cached Files" is about 2 GB in his system, and about 5 on mine. So it seems that recent OSX versions limit the file/dir cache to a percentage of the RAM, whereas old systems used it all when available. Is that so?
>
> Is there a remedy, e.g. a setting to increase the file cache, at least temporarily?
>
> --
> Thomas Tempelmann, http://www.tempel.org/
> Follow me on Twitter: https://twitter.com/tempelorg
> Read my programming blog: http://blog.tempel.org/
> _______________________________________________
> Do not post admin requests to the list. They will be ignored.
> Filesystem-dev mailing list (email@hidden)
> Help/Unsubscribe/Update your Subscription:
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden