Re: FSSetForkSize writing massive amounts of data over network?
Re: FSSetForkSize writing massive amounts of data over network?
- Subject: Re: FSSetForkSize writing massive amounts of data over network?
- From: Jim Luther <email@hidden>
- Date: Thu, 3 Feb 2011 10:05:02 -0800
On Feb 2, 2011, at 9:39 PM, James Bucanek wrote:
>> On Mac OS X, HFS follows the POSIX behavior of requiring that all unwritten
>> ranges of a file must return zeroes when read back.
>
> That's news to me. I guess the documentation is long overdue for an update (I'll file a bug report):
>
> From the FSSetForkSize documentation:
>
> Discussion
> ... If the fork’s new size is greater than the fork’s current size, then the
> additional bytes, between the old and new size, will have an undetermined value.
>
> This lead me to believe that no data was actually written during an FSSetForkSize, and that any new blocks allocated to the file would simply contain the block's previous content.
>
> On further reflection, this behavior is probably a security concern.
"undetermined value" does not mean it won't be written -- it means you shouldn't assume anything about the contents of the newly allocated portion of the file. Don't bother with a bug report because that documentation isn't likely to change.
On Feb 2, 2011, at 9:39 PM, James Bucanek wrote:
>>> Next I tried to use FSAllocateFork on the network volume and it returns
>>> almost instantly, but it also doesn't appear to do anything. Testing the free space on the volume before and after the FSAllocateFork call
>>> returns the same value. Setting the fork size after the FSAllocateFork call
>>> produces the same results as before. So it doesn't appear that anything was
>>> actually allocated by FSAllocateFork.
>>
>> From the documentation for FSAllocateFork:
>>> The FSAllocateFork function attempts to allocate requestCount bytes of
>>> physical storage starting at the offset specified by the positionMode
>>> andpositionOffset parameters. For volume formats that support preallocated
>>> space, you can later write to this range of bytes (including extending the
>>> size of the fork) without requiring an implicit allocation.
>> Notice the "attempts to allocate" wording. HFS is one of the few file systems
>> that is capable of having the physical file size be significantly larger than
>> the logical size (more than just rounding up to a multiple of an allocation
>> block). Most file systems cannot physically allocate extra space without also
>> changing the logical EOF. In the case of the network volume, I would expect
>> the actualCount output to be zero, indicating that it didn't actually allocate
>> any additional physical space.
>
> The actualCount value returned for the network volume is the same as the requested value, indicating a successful allocation. In my test case these were values between 100MB and 160GB. In every test, the free space reported by the volume remained the same.
>
> On the local drive, the actualCount returned was also the same as the requested value, but the free space on the volume dropped accordingly.
That's the way Allocate has worked (except for bugs along the way like 4843619) on AFP for as long as I've worked here (over 20 years). Here's some comments from the old System 6/7/8/9 AFP client (yes, those are 68K assembly comments):
; Those of you who have read the AFP spec know that we don't support this call. So,
; you may ask, what do we do? Well, what we will do is return noErr in all cases
; except when the amount of space asked for is greater than the amount of free space
; we think is on the volume. (Hey, it's the best we can do.)
;
; Note that AllocContig is impossible to support so it's supported just like
; the Allocate call.
On Feb 3, 2011, at 9:16 AM, Mark Day wrote:
> On Feb 2, 2011, at 9:39 PM, James Bucanek wrote:
>
>> The actualCount value returned for the network volume is the same as the requested value, indicating a successful allocation. In my test case these were values between 100MB and 160GB. In every test, the free space reported by the volume remained the same.
>
> That sounds like a bug to me.
No, that's how the Allocate works on file systems that don't support it. Again, from the old AFP client sources... If the space requested was available (which is racy since another client could use that space at any time), it:
;
; Enough space on disk to satisfy allocate call. Return requested number in ioActCount and
; set noErr.
;
On Feb 2, 2011, at 9:39 PM, James Bucanek wrote:
> And this isn't my first run in with FSAllocateFork. It was horribly broken in 10.4, was also broken in 10.5 file sharing, and is still broken in whatever version of AFP ships with the Airport Basestation and Time Capsule (bug #4843619). In that bug, the problem is that FSAllocateFork miscalculates the size of the allocation by always adding the size of the existing file to the allocation. This is moot if the file length is zero, but not if the file already contains data.
I fixed <rdar://problem/4843619> in Leopard.
And yes, I'm guilty of using setting the EOF to allocate space. The old MoreFiles sample (which is deprecated now for reasons such as this) used PBSetEOFSync to preallocate space when copying files. My only defense is I didn't know Mac OS X was coming when I wrote it in the early 1990s.
So, Mark's recommendation, "We've generally advised applications to use FSAllocateFork (or the equivalent at other API layers) to try to reserve the space up front, and then just write to the file when they need to, and handle errors appropriately.", is still my recommendation, too. FSAllocateFork is a hint to attempt to do things more efficiently.
- Jim
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden