I am finding dng image files are corrupting. Canon's cry seem okay. crw image files seem to last. psd files seem to last. Some tiff files are corrupting. Ideas and suggestions are welcome, These are files on a mirrored RAID with three 1 GB HD's. and also on an external HD. As well as some on the computer's internal HD. Actually both computers. Both are MacBook Pro 8,3, Mt Lion, 17inch non-glare display. Both have external monitors. Anyone else having this issue? Cheers David Millers' Photography L.L.C. digitalimaging@dnmillerphoto.com David B. Miller, Pharm. D. member 3809 Alabama Street Bellingham, WA 98226-4585 360 739 2826 Sent from my MacBook Pro 17 inch
oops I meant Canon's cr2 and crw Millers' Photography L.L.C. David Miller, member Spinnaker Photo Imaging Center spinnakerphotoimagingcenter@dnmillerphoto.com 360 739 2826 On Apr 6, 2013, at 2:45 PM, "Millers' Photography L.L.C." <digitalimaging@dnmillerphoto.com> wrote:
I am finding dng image files are corrupting. Canon's cry seem okay. crw image files seem to last.
psd files seem to last. Some tiff files are corrupting.
Ideas and suggestions are welcome,
These are files on a mirrored RAID with three 1 GB HD's. and also on an external HD. As well as some on the computer's internal HD. Actually both computers. Both are MacBook Pro 8,3, Mt Lion, 17inch non-glare display. Both have external monitors.
Anyone else having this issue?
Cheers
David
Millers' Photography L.L.C. digitalimaging@dnmillerphoto.com David B. Miller, Pharm. D. member 3809 Alabama Street Bellingham, WA 98226-4585 360 739 2826 Sent from my MacBook Pro 17 inch
_______________________________________________ Do not post admin requests to the list. They will be ignored. Colorsync-users mailing list (Colorsync-users@lists.apple.com) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/colorsync-users/spinnakerphotoimagin...
This email sent to spinnakerphotoimagingcenter@dnmillerphoto.com
Any Photoshop update recently? 2013/4/6 Spinnaker Photo Imaging Center < spinnakerphotoimagingcenter@dnmillerphoto.com>
oops I meant Canon's cr2 and crw
Millers' Photography L.L.C. David Miller, member Spinnaker Photo Imaging Center spinnakerphotoimagingcenter@dnmillerphoto.com
360 739 2826
On Apr 6, 2013, at 2:45 PM, "Millers' Photography L.L.C." < digitalimaging@dnmillerphoto.com> wrote:
I am finding dng image files are corrupting. Canon's cry seem okay. crw image files seem to last.
psd files seem to last. Some tiff files are corrupting.
Ideas and suggestions are welcome,
These are files on a mirrored RAID with three 1 GB HD's. and also on an external HD. As well as some on the computer's internal HD. Actually both computers. Both are MacBook Pro 8,3, Mt Lion, 17inch non-glare display. Both have external monitors.
Anyone else having this issue?
Cheers
David
Millers' Photography L.L.C. digitalimaging@dnmillerphoto.com David B. Miller, Pharm. D. member 3809 Alabama Street Bellingham, WA 98226-4585 360 739 2826 Sent from my MacBook Pro 17 inch
_______________________________________________ Do not post admin requests to the list. They will be ignored. Colorsync-users mailing list (Colorsync-users@lists.apple.com) Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/colorsync-users/spinnakerphotoimagin...
This email sent to spinnakerphotoimagingcenter@dnmillerphoto.com
_______________________________________________ Do not post admin requests to the list. They will be ignored. Colorsync-users mailing list (Colorsync-users@lists.apple.com) Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/colorsync-users/jbueno61%40gmail.com
This email sent to jbueno61@gmail.com
Adobe Download Manager running on background and don't reset application and computer. 2013/4/7 José Ángel Bueno García <jbueno61@gmail.com>
Any Photoshop update recently?
2013/4/6 Spinnaker Photo Imaging Center < spinnakerphotoimagingcenter@dnmillerphoto.com>
oops I meant Canon's cr2 and crw
Millers' Photography L.L.C. David Miller, member Spinnaker Photo Imaging Center spinnakerphotoimagingcenter@dnmillerphoto.com
360 739 2826
On Apr 6, 2013, at 2:45 PM, "Millers' Photography L.L.C." < digitalimaging@dnmillerphoto.com> wrote:
I am finding dng image files are corrupting. Canon's cry seem okay. crw image files seem to last.
psd files seem to last. Some tiff files are corrupting.
Ideas and suggestions are welcome,
These are files on a mirrored RAID with three 1 GB HD's. and also on an external HD. As well as some on the computer's internal HD. Actually both computers. Both are MacBook Pro 8,3, Mt Lion, 17inch non-glare display. Both have external monitors.
Anyone else having this issue?
Cheers
David
Millers' Photography L.L.C. digitalimaging@dnmillerphoto.com David B. Miller, Pharm. D. member 3809 Alabama Street Bellingham, WA 98226-4585 360 739 2826 Sent from my MacBook Pro 17 inch
_______________________________________________ Do not post admin requests to the list. They will be ignored. Colorsync-users mailing list (Colorsync-users@lists.apple.com) Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/colorsync-users/spinnakerphotoimagin...
This email sent to spinnakerphotoimagingcenter@dnmillerphoto.com
_______________________________________________ Do not post admin requests to the list. They will be ignored. Colorsync-users mailing list (Colorsync-users@lists.apple.com) Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/colorsync-users/jbueno61%40gmail.com
This email sent to jbueno61@gmail.com
Millers' Photography L.L.C. wrote:
I am finding dng image files are corrupting. Ideas and suggestions are welcome,
This may or may not be related to your problem, but a thing that rather flies below most peoples radar is that modern popular computer systems have vast amounts of RAM that is not error checked (parity) or error corrected (ECC). This means that random bit flips caused by alpha rays will happily crash your programs or corrupt your files if the files are loaded into RAM and then written back to disk. [As a H/W engineer designing computer RAM systems in the 80 & 90's, I was rather horrified to learn this. But reducing system costs seems to have won the day over reliability.] These bit flip events are relatively rare with modern RAM chips, but they do happen (a symptom I occasionally saw was a program that would suddenly start crashing for no reason, even after re-loading it. A system re-start would cure the problem. What had happened was that the system disk cache had become corrupted). Intel reserves ECC for its server chipsets (although there is the occassional exception in the laptop space), so you are pretty much stuck with the problem if you choose Intel. AMD has ECC support on many of its chipsets, so it is just a matter of locating and fitting ECC capable RAM SIMs and enabling ECC in the BIOS. I certainly sleep a bit easier now that I switched to an AMD CPU/chipset, and have ECC protecting my data when it's in RAM. Graeme Gill.
Graeme, do you use IBM compatible, PC? Davod Millers' Photography L.L.C. David Miller, member Spinnaker Photo Imaging Center spinnakerphotoimagingcenter@dnmillerphoto.com 360 739 2826 On Apr 6, 2013, at 6:00 PM, Graeme Gill <graeme2@argyllcms.com> wrote:
Millers' Photography L.L.C. wrote:
I am finding dng image files are corrupting. Ideas and suggestions are welcome,
This may or may not be related to your problem, but a thing that rather flies below most peoples radar is that modern popular computer systems have vast amounts of RAM that is not error checked (parity) or error corrected (ECC). This means that random bit flips caused by alpha rays will happily crash your programs or corrupt your files if the files are loaded into RAM and then written back to disk.
[As a H/W engineer designing computer RAM systems in the 80 & 90's, I was rather horrified to learn this. But reducing system costs seems to have won the day over reliability.]
These bit flip events are relatively rare with modern RAM chips, but they do happen (a symptom I occasionally saw was a program that would suddenly start crashing for no reason, even after re-loading it. A system re-start would cure the problem. What had happened was that the system disk cache had become corrupted).
Intel reserves ECC for its server chipsets (although there is the occassional exception in the laptop space), so you are pretty much stuck with the problem if you choose Intel. AMD has ECC support on many of its chipsets, so it is just a matter of locating and fitting ECC capable RAM SIMs and enabling ECC in the BIOS. I certainly sleep a bit easier now that I switched to an AMD CPU/chipset, and have ECC protecting my data when it's in RAM.
Graeme Gill. _______________________________________________ Do not post admin requests to the list. They will be ignored. Colorsync-users mailing list (Colorsync-users@lists.apple.com) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/colorsync-users/spinnakerphotoimagin...
This email sent to spinnakerphotoimagingcenter@dnmillerphoto.com
Millers' Photography L.L.C. wrote:
I am finding dng image files are corrupting.
To give a broader answer, any sort of unchecked soft error in the system(s) data paths can cause this sort of issue. Often a system will seem to run fine, even when there is such an issue in its memory or storage system, but problems will show up with larger data sets. [The very first IBM compat. PC I bought had such a problem. The hard drive was mounted over the motherboard, and itturned out that RFI from the motherboard was corrupting some unchecked data path in the hard drive electronics. I tracked it down by writing some scripts to create large files and then checking their contents. Putting a metal shield over the hard drive electronics fixed that particular problem.] See <http://en.wikipedia.org/wiki/Soft_error> Graeme Gill.
On Apr 6, 2013, at 3:45 PM, Millers' Photography L.L.C. <digitalimaging@dnmillerphoto.com> wrote:
I am finding dng image files are corrupting.
How do you know they're corrupt? DNG checksums are supported only since the implementation of the April 2008, DNG v1.2 spec. Without the checksum it's possible they're corrupted, yet open without error. And with the checksum, it's possible you're notified of corruption, but the corruption is minor (may affect a pixel in a non-visible manner, or may only affect metadata).
Canon's cry seem okay. crw image files seem to last.
psd files seem to last. Some tiff files are corrupting.
How do you know the TIFFs are corrupt? Some file formats are more sensitive to corruption than others. On the one hand a TIFF is more prone to corruption because the file sizes are larger, so there's a bigger footprint for corruption to occur. However, if random corruption occurs with uncompressed TIFF and a JPEG, the TIFF will be less affected than the JPEG because corruption of compressed data has a larger negative effect. So you may just be running into some bad luck with certain file formats, I don't off hand see a pattern.
These are files on a mirrored RAID with three 1 GB HD's. and also on an external HD. As well as some on the computer's internal HD.
This is a three way mirror using Apple software RAID? In any case, the mirror instantly replicates corrupt data in memory to all disks in the mirror. The disk sees the corrupt data as valid as any other data. And it's replicated into your backups. And there's no work around for this except to do important work only on computers with ECC memory, which for some people is a show stopper (including myself). So you just have to accept that some corruption is a possibility. Further it's unclear how Apple software RAID deals with the very long error recovery times in most consumer hard drives, when it comes to repairing bad sectors (drive ECC detected but uncorrectable errors). I've asked a couple times on the filesystem forum but haven't received a response from anyone. So the RAID may be accumulating bad sectors on one or more disk, if they aren't being repaired by the RAID layer.
Both are MacBook Pro 8,3, Mt Lion, 17inch non-glare display. Both have external monitors.
This model can't use ECC memory, so as Graeme mentioned, this can be a source of the initial corruption. Apple's MacPros are the only computers they sell that use ECC memory. But there are many other sources for such corruption between the data on the disk and the data in memory (AHCI, the disk controller, the cable, the drive itself which contains a number of components each one of which can induce corruption silently). Consumer drives experience uncorrectable errors statistically at one order magnitude higher than nearline drives, and two orders magnitude more than enterprise drives, and three to four orders magnitude more errors than tape. It's such a problem that RAID 5 with high density consumer drives is a really bad idea, there's a rather significant chance of losing all data on the array with merely a one disk failure. (Most manufacturers proscribe the use of consumer drives with RAID 5 BTW). And there are user behaviors that induces replication of corruption. e.g. rotation of all back up media, without an archive. And the even worse, but quite common, case of making backups from backups. The dirty secret of digital photography absolutely is the storage stack. Chris Murphy
participants (6)
-
Chris Murphy
-
Graeme Gill
-
Iliah Borg
-
José Ángel Bueno García
-
Millers' Photography L.L.C.
-
Spinnaker Photo Imaging Center