I don't have a blog where I could rant about this so I do this here. It's about general file system performance, so I hope it's not too off-topic.
a) Some background, and an example of how to keep your file-access-intensive programs in high performance:
I recently learned that my app "Find Any File", which is basically a UI for FSCatalogSearch, was very slow when doing recursive searches on network volumes that didn't use AFP, i.e. that didn't support CatSearch.
I eventually realized that this was for the following shortcomings in my code:
(BTW, I used the "fs_usage" tool to see the API calls.)
1. Instead of initially asking only for the least amount of directory attributes for each item I wanted to filter for the search, I asked for too many, causing a lot more traffic on SMB mounts than necessary. E.g, if the user only wanted to search for file names, I should only ask for the file names and the directory flag, but not for all the other attrs such as sizes, dates etc. By using the proper flags with FSGetCatalogInfo(Bulk) I solved this issue.
2. When displaying the found items, I had to make sure that I fetch the remaining information for display (dates, sizes, path etc.) only once in a single call, as each FS call would usually also cause another network call, even if one would think that the data should already be cached on the local side. But it appears that the OS likes to re-check for the existence of the file whenever you poll another information about it, thus making another inquiry over the network. While I can manage to fetch an item's attributes only once, I also need the help of some higher-level OS functions, e.g. to Lauch Services in order to tell if an item is hidden or is a registered bundle. Unfortunately, this OS function makes its own calls to the FS, thus causing another network inquiry for data that I had already fetched. There seems to be no way to prevent this repeated fetching, leading to doubling the network traffic and causing unncessary slowdowns when displaying the results. Lastly, fetching the proper icons for the files again causes a LOT of FS API calls, all leading to a lot more network traffic, with many of those getattrlist calls being performed again and again for the same items. All this should be avoidable, but the current FS APIs do not offer a way to prevent that, unfortunately. Something like a cache mode would be great here, as the information that's going to be fetched is only used for display of data that's partially in showing already anyway. Therefore, having to re-check with the source of the file while fetching the same attributes again should be based on the initially fetched data as that's what the display is being based on.
Now that I think of it, I wonder if an API could be provided for improving this:
On a per-process basis, an item's information could be tied to a cache, so that further inquiries about that item would prefer the cached data, avoiding disk/network access unless that data hasn't been fetched yet.
That should deal with issue #2 above, avoid all those repeated network inquiries for practically the same information.
By binding that cache functionality to a process, other apps that need real-time information about a file won't get affected by stale caches.
In a way this comes close to the CFURL based API, I guess. However, AFAIK, CFURL is still caching only that particular object, but not the underlying disk item it refers to. So, if I call the Launch Services to get information about an item, they'd still fetch their own CFURL object, thereby causing its own network inquiry again, won't they?
I guess something like CFURL, but tied not to the CFURL object but to the path of the file, would be the best solution, i.e. a CFCachedDiskItem class, which locks an disk item (via its path or URL) in the cache as long as the object exists.
Anyhow, that's just a half-baked idea. No rocket-science, either. Just observations.
b) Finder and Time Machine.
I am a big fan of Time Machine, at least on the technical level. I am less a fan of its UI, along with the Finder's, seeing how slow it is especially when viewing network volumes, and how it's seemingly becoming even slower with every OSX upgrade.
I use TM with a local, internal, disk, and it is fairly fast and reliable.
I had once considered getting a Time Capsule, but was suspicious about the fact that it's using a rather slow network connection. But then, Apple sells it as a great solution, so it must be working well, mustn't it?
Well, today, a friend showed me the horrors of Time Capsule: Saving a few 100MB of changed files to the Capsule can easily take one to two hours. Which is incredible, isn't it?
And why is that? I bet it's because of the same sloppy programming that I've been doing myself: Using too many convenience functions that keep fetching the entire set of attributes again and again. And while my program uses the (deprecated?) FSRef based functions that at least let me fetch all attributes in one go, I'd not be surprised if the Finder (and therefore also TM) uses Cocoa APIs where you'd fetch a single attribute by each method call (name, lenth, mod date, creation date) and so on, probably causing even many more FS API calls for the same item. Or don't they? The slowness of opening a remote folder in the Finder would well support that theory, IMHO.
And with just a quick glance at the accesses on the TM volume with fs_usage when performing a TM backup I can see that many many accesses to the SAME item are repeated 10-20 times. Each of them issuing a fresh network call. Simply cutting these down should be able to speed up TM backup to a network by a huge amount.
Now, I wonder why this isn't happening. Isn't anyone at Apple noticing that TC or any network-based TM backup is so incredibly slow?
I was briefly interviewed a few years back by Apple for a position on the team working on file and backup solutions, but I was not accepted (no idea why - but I guess my German directness gave me away as a bad team player) -- Sometimes, like now, I wish I had gotten the chance to be part of that team and help move things forward.
Okay, so what am I getting to?
First, I like to see if this can get anyone's attention at Apple so that they start doing something about it (or, can someone get me in contact with someone who'd care to listen?)
Second, I'm still available for contracting, should Apple want a programmer that digs his teeth into this. I'm good at that. :-D
Lastly, I shared my own experience because I hope other developers are made a little more aware of the issues I described above.
--
Thomas Tempelmann,
http://www.tempel.org/Follow me on Twitter:
http://twitter.com/#!/tempelorg