Re: single hdd can drop xsan performance

I am so insanely happy to report that a drive failed in one of my Xserve RAIDs last night.

Why am I so happy about it? Because I have been dogged by slow read and write speeds on the Video storage pool (4 LUNs) for the last couple of weeks, and HD capture was becoming more and more difficult and then impossible. And when the Xsan is doing badly, all eyes look to me to fix it, of course. When we can't capture HD and we have 4 Xserve RAIDs and 30 seats of Xsan, then it feels like a bad investment.

But thankfully the Xserve RAID emailed me last night reporting: "XSR03, A hard drive SMART pre-failure occurred." What a sigh of relief. After discussing this issue with many on this list, and hearing about drives failing but not reporting that they have failed.... I was told to watch the activity lights on the drives, and watch I did. I suspected that this exact drive that did fail would fail, but it is not exactly a pure science (watching drive lights?!). I thought it might be this drive 8, after watching many writes, but it wasn't 100% certain. And with only one spare, and no RAID error I could not just swap drives that I had a hunch on. Thankfully it emailed me, I replaced it, it was the one I thought would go, and I now I can get a replacement for this bad drive. And more importantly we can capture HD again, hopefully. I have not test the Xsan and HD this morning. The array is rebuilding itself, but I will test throughout the day. And maybe the Xsan will just work again and I can stop taking the heat for this drive#8.

I email this to you all as a follow up to this previous discussion on the matter, and to show that drives do go bad before they report it Here some of the event log:

<snip>

Lower Controller 08/03/06 09:13:46 AM Disk 8 Online

Lower Controller 08/03/06 09:13:37 AM Disk 8 Failed

Upper Controller 08/03/06 09:13:01 AM Error Status Cleared Using Service ID Button

Lower Controller 08/03/06 09:13:01 AM Error Status Cleared Using Service ID Button

Lower Controller 08/03/06 09:12:47 AM Disk 8 Offline

Lower Controller 08/03/06 09:12:47 AM Email Notification Sent (0)

Lower Controller 08/03/06 09:10:42 AM Error Status Cleared Using Service ID Button

Upper Controller 08/03/06 09:10:41 AM Error Status Cleared Using Service ID Button

Lower Controller 08/02/06 09:20:22 PM Email Notification Sent (0)

Lower Controller 08/02/06 09:20:21 PM Disk 8 SMART Failure

<snip>

Mat X

System Administrator

Anthem Visual Effects, Inc.

200 - 110 Cambie Street

Vancouver, BC V6B 2M8

Phone: 604-669-9936

Fax: 604-669-9926

On Jun 29, 2006, at 7:10 AM, Flynn, Daniel wrote:

Hi Xsan users,

There are certain situations when a single disk in a LUN will under
perform considerably and NOT be failed by the XserveRAID. This may be by
design as a conscious philosophy of availability over performance. I
have had this occur now three times, and last night was the first where
it occurred on a disk in an Xsan volume.