Re: problems with hfs_truncatefs() / HFS_RESIZE_VOLUME
Re: problems with hfs_truncatefs() / HFS_RESIZE_VOLUME
- Subject: Re: problems with hfs_truncatefs() / HFS_RESIZE_VOLUME
- From: Michael Alfred Schmidt <email@hidden>
- Date: Tue, 27 Jan 2009 09:07:31 +0100
That's a clear word, thanks very much!
Michael
| Terry Lambert <email@hidden>
27.01.2009 05:01
|
|
To
| Michael Alfred Schmidt <email@hidden>
|
|
cc
| darwinKernel Dev <email@hidden>
|
|
Subject
| Re: problems with hfs_truncatefs() /
HFS_RESIZE_VOLUME |
|
The fsctl() routine is considered SPI - System Private
Interface - for
use bu the diskutil command. It's semantics are subject to more
frequent change that the input and output formats from the diskutil
command. There is a reason we don't document fsctl(), but that we
do
document diskutil.
The problem ID referenced in TN2053 in terms of documentation,
3035842, was closed as "Behaves Correctly, in that we do not intend
to
expose or support the use of the function by third party developers.
Like I said, you'd probably be better off in terms of portability
using popen() or system() to invoke the diskutil program with
appropriate arguments, rather than using this directly. This is a
case of "lower level is not necessarily better".
If you were trying to use sysctl with KERN_PROCARGS2 or something
similar, I'd be telling you to popen() the ps command and give it
known explicit format and argument strings to control what output you
get, for similar reasons: when we rev the OS, or change SPIs in
software updates, we rev the things that depend on them in lock-step
at the same time. Things which conform to standards (like ps) or
which have had stable user interfaces for a long time (like diskutil)
are much less likely to change than the SPIs they defend upon.
Hope that answers your question adequately.
-- Terry
On Jan 26, 2009, at 12:41 AM, Michael Alfred Schmidt wrote:
> Hi Terry,
>
> Thank you very much for cour comprehensive comments that really hit
> the point! With HFSIOC_RESIZE_VOLUME, the code works as expected.
I
> will notify the author of 'Mac OS X Internals' and notify him that
> the code snippet in his book needs to be modified.
>
> I'm just considering your concern about using the 'fsctl ()' kernel
> API call rather than 'diskutil'. For me (just coming from the
> Windows world) the option to use an API call that is documented in
> the open source community looks much more reliable and future-proof
> than calling an executable that might not be present on every
> machine (because somebody may have fiddled around with it) or whose
> calling conventions might change over time. Maybe I'm wrong here.
> What's your take on this?
>
>
> Thanks again,
>
> Michael
>
>
>
> Terry Lambert <email@hidden>
> 23.01.2009 23:40
>
> To
> Michael Alfred Schmidt <email@hidden>
> cc
> Shantonu Sen <email@hidden>, email@hidden
> Subject
> Re: problems with hfs_truncatefs() / HFS_RESIZE_VOLUME
>
>
>
>
>
> You have three problems. The first one is the most severe:
>
> 1)
>
> You should really be solving this problem by using popen() on the
> diskutil command, rather than trying to replace/rewrite the diskutil
> command. If you had started out this way, no matter what changes
we
> made in the future, so long as the input and output of that command
> didn't change, then your code would have been isolated and continued
> to run and be happy. You didn't do this.
>
> 2)
>
> The second problem is that you didn't give us all the information
up
> front. The most important piece you left out for this particular
type
> of problem is that you failed to tell us your code was 64 bit. When
> passing size variant parameters to fsctl, ioctl, and fcntl, it's
> important to know this. Effectively, these end up with different
> definition values for their manifest constants, and when that happens,
> they do not match the case statement entries for the 32 bit values,
so
> unless we've gone out of our way to deal with this issue, many of
> these controls will not work from 32 bit code (size variance can also
> be a function of structure packing, even if the parameter structure
> contains no sized types).
>
> Fortunately for you, the parameter in this case is 64 bit on both
32
> and 64 bit platforms, but...
>
> 3)
>
> Now the third problem: you are using the wrong command parameter to
> accomplish what you are attempting to accomplish.
>
> The command parameter you are using is HFS_RESIZE_VOLUME; the correct
> one is HFSIOC_RESIZE_VOLUME:
>
> #define HFSIOC_RESIZE_VOLUME
_IOW('h', 2, u_int64_t)
> #define HFS_RESIZE_VOLUME
> IOCBASECMD(HFSIOC_RESIZE_VOLUME)
>
> What does IOCBASECMD() do? It strips the command size out of
the
> manifest constant, like so:
>
> #define IOCBASEDCMD(x)
((x) &
> ~(IOCPARM_MASK << 16))
>
> Why? Because when you are calling fsctl, fcntl, or ioctl with
a size
> variant argument value, this will allow you to have a single case
> statement entry which is agnostic as to t whether or not the command
> is coming from 32 or 64 bit code. This is useful, in that it
allows
> us to isolate the bit-ness dependence into the case statement entry.
> This is why when it's decoding these, it applies the IOCBASEDCMD()
> macro to the command argument, and uses the size-agnostic version
of
> the call to do the switch statement case entries, and then these
> entries go on and cast the data portion of the command buffer to
> parameter value pointers of the appropriate type, and dereference
them
> for input and output.
>
> This works this way because the fsctl, fcntl, and ioctl code in the
> kernel copies the parameter data into an internal kernel buffer (size
> constrained) and deal with the data as kernel data rather than user
> data (among other things, this prevents the race between validation
> and use of the data in some cases where it would otherwise have to
be
> copied in twice). When the operation is done, it copies the
data back
> out. Both of these copy operations are predicated on control
bits in
> the command parameter (specifically, if IOC_IN is present, the data
is
> copied in on the way in and if IOC_OUT is present, it's copied out
on
> the way out).
>
> So given that you made the call and told it that your data length
is
> 0, how much data was copied in? 0 bytes. So what's in
the buffer
> 'data' that's getting dereferenced out? Stack garbage. This
is why
> the value is the same each time you call it. Luckily, since
these are
> data interfaces behind an authorization check, so you don't have to
> worry about this being a security issue; the data being dereferenced
> is already in a kernel buffer, so it's not gong to result in a fault
> on reference, and since it's no procedural, as long as we range check
> our arguments (we do), then you're not going to do anything much
> (except get yourself an error, which is what you saw was happening).
>
> --
>
> My suggestion would be to not attempt to use non-public interfaces
> unless you understand precisely how they work; even the public
> interfaces, as problem #2 demonstrates, can be problematic. Your
best
> bet would be to wrap system tools which do what you want, and are
well
> enough documented and mature enough, that you aren't going to have
to
> worry much about them changing on you.
>
> If you insist on going forward in writing your own low level code
for
> this instead of writing intermediate code to wrap the utility, well,
> at least now you know why it was failing, why you were seeing the
> results you were seeing, and to be more careful about research in
the
> future.
>
> Cheers,
> -- Terry
>
>
> On Jan 23, 2009, at 7:28 AM, Michael Alfred Schmidt wrote:
> > Thanks, I understand. Anyway, my goal is to shrink the volume
rather
> > than to expand it. So the repartitioning problem shouldn't apply.
> >
> > As I eventually managed to set up the kernel debugger, I was
able to
> > verify the calling parameters of 'hfs_truncatefs()' (in the kernel
-
> > hfs_vfsops.c). What I figured there is that the 'newsize' paramter
> > always had a constant value, regardless of the various valid
input
> > parameters that I passed into userland 'fsctl()' for
> > 'uiNewSize' (see my code). So I assume that this value gets lost
> > somewhwere between userland and kernel. This is why I assume
that I
> > might have some kind of thunking problem (maybe between 32-bit/64-
> > bit code) with my input pointer. Unfortunately, my kernel debugging
> > skills aren't good enough yet to trace the parameter passing
to the
> > kernel.
> >
> > Thanks anyway so far!
> >
> > Michael
> >
> >
> >
> > Shantonu Sen <email@hidden>
> > 23.01.2009 15:52
> >
> > To
> > Michael Alfred Schmidt <email@hidden>
> > cc
> > email@hidden
> > Subject
> > Re: problems with hfs_truncatefs() / HFS_RESIZE_VOLUME
> >
> >
> >
> >
> >
> > Please read my post again.
> >
> > diskutil is an integrated solution that does both repartitioning
and
> > filesystem resizing. The "limits" it prints out assume
that it is
> > also going to be doing repartitioning.
> >
> > The 128MB freespace is outside of the partition boundaries. If
you
> > want to use it (which I don't recommend), you need to repartition
> > first to increase the size of your partition to use some of that
> > freespace, and then increase the filesystem to match.
> >
> > Shantonu
> >
> > Sent from my MacBook
> >
> > On Jan 23, 2009, at 12:25 AM, Michael Alfred Schmidt wrote:
> >
> >
> > Shantonu,
> >
> > I'm testing on a MacBook Pro with default factory partitioning,
i.e.
> > there are 128 MB left free on top of the OS partition (see http://developer.apple.com/technotes/tn2006/tn2166.html#SECPARTITIONINGPOLICY)
> > . I verify upfront with 'diskutil resizeVolume <resp. volume
device>
> > limits' what freedom to expand or shrink I have. Since my goal
is to
> > shrink the partition rather than to expand it anyway, I concentrate
> > my tests on shrinking (by ~ 10%).
> >
> > Michael
> >
> >
> > Shantonu Sen <email@hidden>
> > 22.01.2009 18:11
> >
> >
> > To
> > Michael Alfred Schmidt <email@hidden>
> > cc
> > mm w <email@hidden>, email@hidden
> > Subject
> > Re: problems with hfs_truncatefs() / HFS_RESIZE_VOLUME
> >
> >
> >
> >
> >
> >
> >
> > How do you know it has space to expand? You know you have to
> > repartition the disk first, right (so that the container block
> > device is large enough).
> >
> > This fsctl is something that is straightforward to use the same
way
> > the Disk Utility/diskutil use it. How big is your filesystem?
How
> > many allocation blocks? What sizes are you requesting?
> >
> > Shantonu Sen
> > email@hidden
> >
> > Sent from my Mac Pro
> >
> > On Jan 22, 2009, at 9:06 AM, Michael Alfred Schmidt wrote:
> >
> >
> > I do get ENOSPC in case the code runs into 'hfs_extendfs(): not
> > enough space on device', or EINVAL in case the code runs into
> > 'hfs_truncatefs(): invalid size'. I get both erros with my test
> > program when I only try often enough, i. e. I get both errors
for
> > the same value. The same behavior (both errors possible) applies
to
> > any other arbitrary value of uiNewSize. This is why I believe
that
> > the value is currently somehow lost.
> >
> > The volume has both sufficient site to expand and to shrink.
> >
> > Michael
> >
> >
> >
> > mm w <email@hidden>
> > 22.01.2009 17:50
> >
> >
> > To
> > Michael Alfred Schmidt <email@hidden>
> > cc
> > email@hidden
> > Subject
> > Re: problems with hfs_truncatefs() / HFS_RESIZE_VOLUME
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > you certainly get a ENOSPC message somewhere, bad sample you
are
> using
> >
> > - yoda
> >
> > Cheers!
> >
> > On Thu, Jan 22, 2009 at 8:16 AM, Michael Alfred Schmidt
> > <email@hidden> wrote:
> > >
> > > Hi,
> > >
> > > I'm trying to shrink an HFS+ partition, just as 'diskutil
> > resizeVolume...'
> > > does. However, I'm facing problems implementing the
> > HFS_RESIZE_VOLUME
> > > command via fsctl(), as proposed by Amit Singh on p. 1572
of 'Mac
> > OS X
> > > Internals'.
> > >
> > > My code looks as follows:
> > >
> > > int iRet = 0;
> > >
> > > u_int32_t uiBlockSize;
> > > u_int64_t uiBlockCount, uiCurrentSize, uiNewSize;
> > >
> > > struct statfs StatFS;
> > >
> > > // only works if device mounted
> > > iRet = statfs ("/etc/", &StatFS);
> > > if (iRet < 0)
> > > {
> > > perror ("statfs failed
with");
> > >
> > > return;
> > > }
> > >
> > > uiBlockSize = StatFS.f_bsize;
> > > uiBlockCount = StatFS.f_blocks;
> > >
> > >
> > > printf ("block size:\t %u\n", uiBlockSize);
> > > printf ("block count:\t %llu\n", uiBlockCount);
> > >
> > >
> > > uiCurrentSize = uiBlockSize * uiBlockCount;
> > > printf ("partition size:\t %llu\n\n", uiBlockSize
* uiBlockCount);
> > >
> > > uiNewSize = uiCurrentSize; // only test here!!!
> > > printf ("new volume size: %llu, hex: %llX\n",
uiNewSize,
> uiNewSize);
> > >
> > >
> > > iRet = fsctl ("/", HFS_RESIZE_VOLUME, &uiNewSize,
0);
> > > if (iRet < 0)
> > > {
> > > perror ("fsctl failed with");
> > >
> > > return;
> > > }
> > >
> > >
> > > fsctl() fails with either 'hfs_extendfs(): not enough space
on
> > device' or
> > > 'hfs_truncatefs(): invalid size', even if the new size equals
the
> > old size
> > > (as in the code sample above). It looks like the &uiNewSize
input
> > parameter
> > > is fully ignored, as other "new sizes" show the
same unpredictable
> > results.
> > > On the kernel side, the respective code resides in bsd/hfs/
> > hfs_readwrite.c.
> > > >From looking at that code, I cannot see any obvious
mistake in my
> > code. I'm
> > > wondering whether there is any thunking (pointer conversion)
> > problem with
> > > the pointer to u_int64_t in the transition from userland
to
> kernel.
> > >
> > > My target system is Mac OS X 10.5.x (Intel), compiled with
XCode
> > 3.1 with
> > > default build settings.
> > >
> > >
> > > I'm grateful for any idea on what's wrong with my code!
> > > Thanks in advance!
> > >
> > > Michael
> > >
> > >
> > > _______________________________________________
> > > Do not post admin requests to the list. They will be ignored.
> > > Darwin-kernel mailing list (email@hidden)
> > > Help/Unsubscribe/Update your Subscription:
> > >
> > > This email sent to email@hidden
> > >
> >
> >
> >
> > --
> > -mmw
> >
> >
> > _______________________________________________
> > Do not post admin requests to the list. They will be ignored.
> > Darwin-kernel mailing list (email@hidden)
> > Help/Unsubscribe/Update your Subscription:
> @apple.com
> >
> > This email sent to email@hidden
> >
> >
> > _______________________________________________
> > Do not post admin requests to the list. They will be ignored.
> > Darwin-kernel mailing list (email@hidden)
> > Help/Unsubscribe/Update your Subscription:
> >
> > This email sent to email@hidden
>
>
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Darwin-kernel mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden