Re: reading/writing files from mounted device directly
Re: reading/writing files from mounted device directly
- Subject: Re: reading/writing files from mounted device directly
- From: Michael Smith <email@hidden>
- Date: Wed, 30 Aug 2006 12:49:59 -0700
On Aug 30, 2006, at 12:03 PM, email@hidden wrote:
I am developing a product for the Macintosh platform that
requires I read and write files from a mounted device.
David,
It's good that you've attempted to get help in this phase of your
development,
as perhaps there is still some hope of fixing the situation.
First, though, I need to understand what you're saying above. If the
product
is in development, why is it "required" that you do this?
If there is some part of the product that is set in stone, one would
assume that
is because it's already committed for some other environment. What
other
environment is that, and how do they solve the problem? Have they
actually
solved it, or do they just wave their hands at it?
If your product is, however, truly still "in development", it would
be wise of
you to reconsider this 'sharing' of a filesystem. There are almost
unquestionably better ways to do what you're trying to do.
Information on
the mounted device is being updated in real time but I am unable to
detect the changes due to the operating system (Darwin?) caching the
contents of the device into a buffer which remains static until I
unmount the drive with a "diskutil unmount" command. Continually
mounting and unmounting the device to see changes in a file is not
desirable.
Certain clues suggest to me that this is probably a FAT filesystem.
Like
most other single-owner filesystems, FAT implementations reasonably
expect that nothing on the disk is going to change unless they change
it, and so they consider themselves free to cache the contents of
various
metadata structures (and file data) as necessary.
Darwin does not cache block devices, period. Filesystems cache
metadata,
and the VM subsystem caches file data. Due to the above consideration
and a basic requirement for performance, this behaviour is not
conditional.
There is no fundamental utility in not caching metadata for read
operations
(one can argue write-through caching aids in disaster recovery, and
many filesystems support this as an option).
FAT, along with most other single-owner filesystem formats, is poorly
suited to use as a sharing format. If you were in total control of
the writer,
and could be assured that there was only ever *one* writer, then it
would be possible to enforce strict ordering rules over which metadata
items were written in which order, and make it possible for a likewise
customised reader to avoid becoming confused.
Such writers and readers would, however, be very poorly-performing
general-purpose FAT implementations. Merely "not caching" the disk
blocks would in no wise be sufficient to achieve this.
Using the IOKit to obtain a handle to the device and then
using POSIX funtions to open/read/write does allow access to the raw
disk, but I can only access bytes. Reading the entire device and
programmatically reproducing the file structure to locate the file
I am
interested in reading is prohibitive.
... yet this is exactly what you expect the filesystem to do.
Is there a no-cache option on mounting what is
effectively a
USB storage device? Or how do I bypass the cached buffer?
As noted above, this is not a practical solution.
Assuming, for the moment, that you are stuck with a device that thinks
that it's reasonable for you to read its active FAT filesystem, you need
to do something like this:
1) Document the strict order of operations as performed by the device
when creating, appending to, shrinking, moving and deleting files.
If the
device performs any sort of deferred write buffering or re-orders or
clusters
its operations, you may want to give up now.
2) Document the metadata items that are required to be read (bootblock,
FAT1/FAT2, root directory, subdirectory, etc.) for all of the file
operations
that you need to perform.
3) Determine an order of reading for these metadata items interleaved
in any fashion amongst the write operations for the procedures in step
1 that *guarantees* that your operations will either succeed or
detectably fail. This needs to include data read/write operations (i.e.
file data must be guaranteed to be on-disk before any metadata
referencing it is committed to disk).
4) Implement the necessary portions of the read-only filesystem in
your application. Note that for FAT, there are plenty of simple,
read-only
implementations that you can lift; consider for example the dosfs.[ch]
from the FreeBSD bootloader that can be found here:
http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libstand/
5) Test. Extensively. The issue you are attempting to resolve with
all this talk of "strict ordering" is that FAT (and most other
filesystems)
have relational dependencies between various metadata items. For
example, the starting cluster number in a directory entry is not useful
for any file longer than a cluster unless the FAT chain in one or both
FATs is valid. If the writing filesystem writes the directory entry
before
it updates the FAT, for example (this is common) then you cannot
read the file (and may become confused trying to follow an invalid
chain, depending on the writing filesystem's policy regarding cleaning
out chains for deleted files). Because these issues are sensitive to
the relative timing of operations between the reader and writer, they
are hard to reproduce and hard to debug. However, in any
un-synchronised system they will eventually manifest and make
your customers unhappy.
At any point during the above procedure, optionally take a blunt object
and beat on the designer of the USB device. Repeat as required.
Local filesystems are *not* multi-client databases. Period.
= Mike
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Darwin-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden