Re: dng image file corrupting
Re: dng image file corrupting
- Subject: Re: dng image file corrupting
- From: Chris Murphy <email@hidden>
- Date: Sun, 28 Apr 2013 09:39:16 -0600
On Apr 6, 2013, at 3:45 PM, Millers' Photography L.L.C. <email@hidden> wrote:
> I am finding dng image files are corrupting.
How do you know they're corrupt?
DNG checksums are supported only since the implementation of the April 2008, DNG v1.2 spec. Without the checksum it's possible they're corrupted, yet open without error. And with the checksum, it's possible you're notified of corruption, but the corruption is minor (may affect a pixel in a non-visible manner, or may only affect metadata).
> Canon's cry seem okay. crw image files seem to last.
>
> psd files seem to last. Some tiff files are corrupting.
How do you know the TIFFs are corrupt?
Some file formats are more sensitive to corruption than others. On the one hand a TIFF is more prone to corruption because the file sizes are larger, so there's a bigger footprint for corruption to occur. However, if random corruption occurs with uncompressed TIFF and a JPEG, the TIFF will be less affected than the JPEG because corruption of compressed data has a larger negative effect. So you may just be running into some bad luck with certain file formats, I don't off hand see a pattern.
> These are files on a mirrored RAID with three 1 GB HD's. and also on an external HD. As well as some on the computer's internal HD.
This is a three way mirror using Apple software RAID?
In any case, the mirror instantly replicates corrupt data in memory to all disks in the mirror. The disk sees the corrupt data as valid as any other data. And it's replicated into your backups. And there's no work around for this except to do important work only on computers with ECC memory, which for some people is a show stopper (including myself). So you just have to accept that some corruption is a possibility.
Further it's unclear how Apple software RAID deals with the very long error recovery times in most consumer hard drives, when it comes to repairing bad sectors (drive ECC detected but uncorrectable errors). I've asked a couple times on the filesystem forum but haven't received a response from anyone. So the RAID may be accumulating bad sectors on one or more disk, if they aren't being repaired by the RAID layer.
> Both are MacBook Pro 8,3, Mt Lion, 17inch non-glare display. Both have external monitors.
This model can't use ECC memory, so as Graeme mentioned, this can be a source of the initial corruption. Apple's MacPros are the only computers they sell that use ECC memory.
But there are many other sources for such corruption between the data on the disk and the data in memory (AHCI, the disk controller, the cable, the drive itself which contains a number of components each one of which can induce corruption silently). Consumer drives experience uncorrectable errors statistically at one order magnitude higher than nearline drives, and two orders magnitude more than enterprise drives, and three to four orders magnitude more errors than tape. It's such a problem that RAID 5 with high density consumer drives is a really bad idea, there's a rather significant chance of losing all data on the array with merely a one disk failure. (Most manufacturers proscribe the use of consumer drives with RAID 5 BTW).
And there are user behaviors that induces replication of corruption. e.g. rotation of all back up media, without an archive. And the even worse, but quite common, case of making backups from backups.
The dirty secret of digital photography absolutely is the storage stack.
Chris Murphy
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Colorsync-users mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden