Re: FSSetForkSize writing massive amounts of data over network?
Re: FSSetForkSize writing massive amounts of data over network?
- Subject: Re: FSSetForkSize writing massive amounts of data over network?
- From: James Bucanek <email@hidden>
- Date: Thu, 3 Feb 2011 17:45:34 -0700
Jim Luther <mailto:email@hidden> wrote (Thursday, February
3, 2011 11:05
AM -0800):
On Feb 2, 2011, at 9:39 PM, James Bucanek wrote:
Next I tried to use FSAllocateFork on the network volume and it returns
almost instantly, but it also doesn't appear to do anything. Testing the
free space on the volume before and after the FSAllocateFork call
returns the same value. Setting the fork size after the FSAllocateFork
call produces the same results as before. So it doesn't appear that
anything was actually allocated by FSAllocateFork.
From the documentation for FSAllocateFork:
The FSAllocateFork function attempts to allocate requestCount bytes of
physical storage starting at the offset specified by the positionMode
andpositionOffset parameters. For volume formats that support preallocated
space, you can later write to this range of bytes (including extending the
size of the fork) without requiring an implicit allocation.
Notice the "attempts to allocate" wording. HFS is one of the few file
systems that is capable of having the physical file size be significantly
larger than the logical size (more than just rounding up to a multiple of
an allocation block). Most file systems cannot physically allocate extra
space without also changing the logical EOF. In the case of the network
volume, I would expect the actualCount output to be zero, indicating that
it didn't actually allocate any additional physical space.
The actualCount value returned for the network volume is the same as the
requested value, indicating a successful allocation. In my
test case these were values between 100MB and 160GB. In every
test, the free
space reported by the volume remained the same.
On the local drive, the actualCount returned was also the same as the
requested value, but the free space on the volume dropped accordingly.
That's the way Allocate has worked (except for bugs along the way like
4843619) on AFP for as long as I've worked here (over 20 years). Here's some
comments from the old System 6/7/8/9 AFP client (yes, those are 68K assembly
comments):
; Those of you who have read the AFP spec know that we don't support this
call. So, ; you may ask, what do we do? Well, what we will do is return
noErr in all cases ; except when the amount of space asked for is greater
than the amount of free space ; we think is on the volume. (Hey, it's the
best we can do.) ; ; Note that AllocContig is impossible to support so it's
supported just like ; the Allocate call.
Thanks for the peek inside the inner working of AFP, but I would
still consider this a bug (on the client end of things).
On Feb 3, 2011, at 9:16 AM, Mark Day wrote:
On Feb 2, 2011, at 9:39 PM, James Bucanek wrote:
The actualCount value returned for the network volume is the same as the
requested value, indicating a successful allocation. In my
test case these were values between 100MB and 160GB. In every test, the free
space reported by the volume remained the same.
That sounds like a bug to me.
No, that's how the Allocate works on file systems that don't support it.
Again, from the old AFP client sources... If the space requested was available
(which is racy since another client could use that space at any time), it:
; ; Enough space on disk to satisfy allocate call. Return requested number in
ioActCount and ; set noErr. ;
If it isn't supported, then the call should report that it did nothing.
If the server doesn't support an Allocate function (either
locally or in its protocol), then the FSAllocateFork function
should return 0 for actualCount.
I understand that not every file server and volume format
supports every feature, and I also understand the FSAllocateFork
isn't _required_ to allocate that space--it might allocate less,
more, or nothing at all.
But if a feature isn't supported then it should be either
emulated or it should report that nothing happened, either
through an error code or, in the case of FSAllocateFork, by
returning 0 for the number of bytes allocated. Only then can the
client software make intelligent decisions.
As it stands now, FSAllocateFork says "Yes, I pre-allocated
those 400 million bytes you requested" when if fact it did nothing.
On Feb 2, 2011, at 9:39 PM, James Bucanek wrote:
And this isn't my first run in with FSAllocateFork. It was horribly broken
in 10.4, was also broken in 10.5 file sharing, and is still
broken in whatever version of AFP ships with the Airport
Basestation and Time
Capsule (bug #4843619). In that bug, the problem is that FSAllocateFork
miscalculates the size of the allocation by always adding the size of the
existing file to the allocation. This is moot if the file length is zero, but
not if the file already contains data.
I fixed <rdar://problem/4843619> in Leopard.
Thank you! I didn't know I had you to thank for that. Now how
can we get the guys who ship the Airport Basestations to update
their code too? ;)
It's also very weird that you're telling me that preallocation
on a AFS server is essentially a noop, because I'm sure I've
bumped into one of the FSAllocateFork bugs on both Airport
basestations and older OS X systems running personal
filesharing. If FSAllocateFork is a noop for networked volumes,
then none of these other devices should have problems ... or am
I missing something?
So, Mark's recommendation, "We've generally advised applications to use
FSAllocateFork (or the equivalent at other API layers) to try to reserve the
space up front, and then just write to the file when they need to, and handle
errors appropriately.", is still my recommendation, too. FSAllocateFork is a
hint to attempt to do things more efficiently.
Well, "handle errors appropriately" is my real dilemma. My
problem is that I'm writing backup software, and it tries very
very very very hard NOT to get into a situation where backup
data store get corrupted and can't be used.
Running out of disk space while the backup store is growing is a
serious problem. If there's critical data that has accumulated
in RAM, and there's no place to write that data, then the backup
data store is totally hosed. I go to some lengths to protect
against this situation by pre-allocating enough space to make a
graceful completion of the backup store possible in all
situations. But if FSAllocateFork isn't actually doing anything ...
--
James Bucanek
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden