“How are we not monitoring for these types of failures, and how when drives are hot swappable and mirrored do we have problems?<\/em>“<\/p>\n The underlying message was clear, since a server had problems due to a disk failure, we must not be doing things right.<\/p>\n First, we\u2019ll start where the skies are bright, and the sales reps smile.<\/p>\n Proper drive mirroring combined with hot-swappable drives prevents drive errors from taking down systems.<\/p>\n The salesman will nod and assure you that with THEIR system, you can get 100 9\u2019s<\/strong> of uptime for the low low price of a large check.<\/p>\n The brochure is always awesome, so I won\u2019t elaborate further.<\/p>\n I believe I have a unicorn promised to me in brochure-land. I hope they are feeding it while I\u2019m here.<\/p>\n Drives are paired up and use mirroring to mitigate dying drives.<\/p>\n We have monitoring detect the bad drives, and quickly replace drives which are identified as going bad.<\/p>\n In Practice, Theory holds up a huge preponderance of the time.<\/p>\n We hot swap and replace bad drives quite often with few ill results.<\/p>\n But, and there is always a \u2018but\u2019.<\/strong><\/em><\/p>\n Drives don\u2019t always die in a graceful fashion. Most of the time, they go quietly into that binary-speckled good night, but sometimes we get the theater production of:<\/p>\n On most servers, the drives share the same communication bus, typically a scsi bus. Some drives die in a slow agonizing fashion, will still take some requests, but will be slow in responding, or reset the bus often. Systems having this type of failure can sometimes be seen \u2018glitching\u2019. Running fine for many things, but inexplicably hanging for brief periods. Other times, there will be no glitching but a series of errors in the log files.<\/p>\n When properly identified, drive replacements tend to go smoothly on these systems.<\/p>\n In this case, the drive dying will foul the scsi bus, preventing the good drives sharing the bus from communicating properly.<\/p>\n This is the scenario we faced with our server that got director level attention.<\/p>\n Drive\u00a0 #2 on the system was going bad and squawking on the bus so much it was causing bus resets, and preventing our properly mirrored OS drives(drives 0,1) from doing their jobs.<\/p>\n A removal of the bad drive #2 allowed drives 0,1 to continue their good work.<\/p>\n This can be the hardest to diagnose of all drive failures. The system starts glitching and complaining about a drive going bad.<\/p>\n We\u2019ll call the bad drive \u201cBad disk 1\u201d<\/p>\n System logs are filled with tales of \u201cBad Disk 1\u201d doing many bad bad<\/strong> things, failed reads, writes, scsi bus timeouts, etc.<\/p>\n But, it could be that supposedly \u201cGood drive 0\u201d is head-faking and fouling the bus so hard, it looks like \u201cBad disk 1\u201d is going bad.<\/p>\n In reality, it\u2019s drive 0 which has gone south on us.<\/p>\n We have had a few cases where we removed the indicated bad drive, only to find out the remaining \u2018good\u2019 drive left in the system is useless.\u00a0 In this case, we swap drives and retry.<\/p>\n There is no known heuristic to differentiate between the \u201cSquawking shrieking death of a drive\u201d and \u201cThe head-faking death of a drive\u201d<\/p>\n This head-faking death of a drive scenario is the reason we keep the indicated bad drives we remove from systems around for a few days.<\/p>\n Drive failures of the above two cases are by and large thankfully rare.<\/p>\n <\/p>\n <\/p>\n","protected":false},"excerpt":{"rendered":" I was asked to explain drive failures and mitigation efforts to some Director level and Aboves relative to our servers, after one server had issues due to a bad drive. “How are we not monitoring for these types of failures, … Continue reading The Brochure<\/h2>\n
In Theory<\/h2>\n
In Practice<\/h2>\n
Lingering death of a drive<\/h2>\n
Squawking shrieking death of a drive<\/h2>\n
The head-faking death of a drive<\/h2>\n