site_archiver@lists.apple.com Delivered-To: darwin-dev@lists.apple.com On Aug 30, 2006, at 12:03 PM, darwin-dev-request@lists.apple.com wrote: I am developing a product for the Macintosh platform that requires I read and write files from a mounted device. David, Information on the mounted device is being updated in real time but I am unable to detect the changes due to the operating system (Darwin?) caching the contents of the device into a buffer which remains static until I unmount the drive with a "diskutil unmount" command. Continually mounting and unmounting the device to see changes in a file is not desirable. Such writers and readers would, however, be very poorly-performing general-purpose FAT implementations. Merely "not caching" the disk blocks would in no wise be sufficient to achieve this. ... yet this is exactly what you expect the filesystem to do. As noted above, this is not a practical solution. Assuming, for the moment, that you are stuck with a device that thinks that it's reasonable for you to read its active FAT filesystem, you need to do something like this: 3) Determine an order of reading for these metadata items interleaved in any fashion amongst the write operations for the procedures in step 1 that *guarantees* that your operations will either succeed or detectably fail. This needs to include data read/write operations (i.e. file data must be guaranteed to be on-disk before any metadata referencing it is committed to disk). At any point during the above procedure, optionally take a blunt object and beat on the designer of the USB device. Repeat as required. Local filesystems are *not* multi-client databases. Period. = Mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.appl... It's good that you've attempted to get help in this phase of your development, as perhaps there is still some hope of fixing the situation. First, though, I need to understand what you're saying above. If the product is in development, why is it "required" that you do this? If there is some part of the product that is set in stone, one would assume that is because it's already committed for some other environment. What other environment is that, and how do they solve the problem? Have they actually solved it, or do they just wave their hands at it? If your product is, however, truly still "in development", it would be wise of you to reconsider this 'sharing' of a filesystem. There are almost unquestionably better ways to do what you're trying to do. Certain clues suggest to me that this is probably a FAT filesystem. Like most other single-owner filesystems, FAT implementations reasonably expect that nothing on the disk is going to change unless they change it, and so they consider themselves free to cache the contents of various metadata structures (and file data) as necessary. Darwin does not cache block devices, period. Filesystems cache metadata, and the VM subsystem caches file data. Due to the above consideration and a basic requirement for performance, this behaviour is not conditional. There is no fundamental utility in not caching metadata for read operations (one can argue write-through caching aids in disaster recovery, and many filesystems support this as an option). FAT, along with most other single-owner filesystem formats, is poorly suited to use as a sharing format. If you were in total control of the writer, and could be assured that there was only ever *one* writer, then it would be possible to enforce strict ordering rules over which metadata items were written in which order, and make it possible for a likewise customised reader to avoid becoming confused. Using the IOKit to obtain a handle to the device and then using POSIX funtions to open/read/write does allow access to the raw disk, but I can only access bytes. Reading the entire device and programmatically reproducing the file structure to locate the file I am interested in reading is prohibitive. Is there a no-cache option on mounting what is effectively a USB storage device? Or how do I bypass the cached buffer? 1) Document the strict order of operations as performed by the device when creating, appending to, shrinking, moving and deleting files. If the device performs any sort of deferred write buffering or re-orders or clusters its operations, you may want to give up now. 2) Document the metadata items that are required to be read (bootblock, FAT1/FAT2, root directory, subdirectory, etc.) for all of the file operations that you need to perform. 4) Implement the necessary portions of the read-only filesystem in your application. Note that for FAT, there are plenty of simple, read-only implementations that you can lift; consider for example the dosfs.[ch] from the FreeBSD bootloader that can be found here: http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libstand/ 5) Test. Extensively. The issue you are attempting to resolve with all this talk of "strict ordering" is that FAT (and most other filesystems) have relational dependencies between various metadata items. For example, the starting cluster number in a directory entry is not useful for any file longer than a cluster unless the FAT chain in one or both FATs is valid. If the writing filesystem writes the directory entry before it updates the FAT, for example (this is common) then you cannot read the file (and may become confused trying to follow an invalid chain, depending on the writing filesystem's policy regarding cleaning out chains for deleted files). Because these issues are sensitive to the relative timing of operations between the reader and writer, they are hard to reproduce and hard to debug. However, in any un-synchronised system they will eventually manifest and make your customers unhappy. This email sent to site_archiver@lists.apple.com