Rebuilds are the riskiest operations that an array can perform; they read each of the members (surviving) from one end to the other and write to the new disk. Therefore, when a rebuild fails, you should ask yourself “how much did it change on the way?” rather than “why did it fail?”
Each time a rebuild runs, it reads all of the existing members in their entirety and writes to the array during execution. On disks with pre-existing failures or weaknesses, it’s likely to be the most probable failure point, and also the point where incorrect data will be written.
In general terms when using an array you are reading fragmented blocks of data. During a rebuild operation every sector of each remaining component will be read from start to finish with no pause. As a result, it may take many hours to complete a rebuild of a large array, depending on the size of the disks; this is the first time in the array’s existence where all parts of all disks have been used at once. Here is also where latent bad sectors will begin to show themselves that were never known before as well as one unreadable sector on a survivor could either halt the rebuild completely or terminate the rebuild operation altogether.
The reason we see so many disk failures during a rebuild of an array versus failure prior to a rebuild is because the disks’ wear occurs collectively, as they tend to be the same age, model and batch. The surviving disks from this group have generally accumulated similar usage and fatigue, as compared to the failed disk.
If the parity can’t be read clearly during the rebuild process, it’s possible for a bad write to occur on the new drive, and this write will contain error(s). The array now states that it is good but contains areas of corrupted data; hence the files appear correct when you list them, however they are unable to open. On some controllers an aborted rebuild also leaves the metadata inconsistent, so the array can no longer be assembled at all.
A failed rebuild will not result in “undo” as some portion of your original configuration has changed. This is why if a rebuild fails, you should not try again to rebuild.
Frequently more than most people think. Each member (including the original bad drive, if you still have it) is completely copied in read-only format. The reason the original drive is important: it contains its stripes prior to the rebuild and where the rebuild was wrong, the original disk may still contain what is right. Once the members are created from the copies, they are rebuilt into an array using the data rather than trusting controllers whose behaviour you now know to be unreliable.
This is the main reason to keep the original disk that was pulled out once a rebuild fails. The amount of data recovered will often be greatly improved by keeping this disk.
As soon as you notice degradation within your RAID, always follow these steps: back up everything you have access to while the array is still readable; replace the bad drive; and lastly, rebuild. Most people’s natural instinct with a RAID failure is to perform a rebuild first. While this may seem like the fastest way to restore functionality, rebuilding a RAID first places the greatest possible stress on your remaining drives which have already lost all of their redundancy.
Treat a RAID failure as a “recovery” if any of the working members show either “reallocation” status or SMART errors. We charge from £500 +VAT for all RAID and server recoveries but offer a completely free 48 hour diagnostic along with a formal written quote.
It usually happens when a surviving drive contains faulty sectors. The rebuild goes through each and every sector of all remaining drives (which often takes hours) and this is the first time an entire array has run “end-to-end”; therefore latent failures will show up at this point. One unreadable sector on a survivor can halt the rebuild process entirely.
If the rebuild cannot read parity correctly it may write incorrect data to the new drive; subsequently the array will report as healthy but contain invalid areas. Files will appear normal but they will be unable to open. Sometimes an unsuccessful rebuild leaves the array’s metadata inconsistent.
Yes, frequently, and more frequently than many people realise. All members are copied read-only and the array is rebuilt from the copies. In addition, if you still possess the original failing drive it should be sent in too: it retains a copy of its stripe(s) prior to the rebuild and can provide valid data where the rebuild provided invalid data.
No. If a second attempt to write on the array is made (and if you’re writing to an exhausted set of disks), it’s probably going to be the second attempt that takes away what little margin there is left. Shut off your array, keep your disks in their bays in original order, and keep the disk you took out.
First back up anything that is still readable. Then install your replacement drive. Lastly, rebuild the array. Rebuilding your array before you secure the data is the most common “instinct” for many people and also the riskiest approach, since a degraded RAID has no redundancy and rebuilding the RAID is always the maximum load it can take on.