A QNAP which has stopped showing a volume is normally recoverable. This is because in most such cases the disks themselves are still fully functional but have lost the storage pool metadata. The task is to read each disk and rebuild the volume outside of the QNAP.
A QNAP provides two layers: storage pools and often thin provisioning (on top of the RAID), therefore it is possible for a volume to “vanish” while each disk within remains fully functional. The process of recovering this involves rebuilding these layers one by one using read-only images.
An individual QNAP volume is made up of three levels. At the bottom there is usually an mdadm Linux software RAID (Redundant Array of Inexpensive Disks); this is where you find your actual data. Above that is a storage pool (usually LVM), and finally, a volume (often thin-provisioned to allow it to dynamically increase). Therefore, if something goes wrong at any level within these 3 layers, all will appear to have failed, leaving all disks intact. That’s why many QNAP jobs we receive show all drives passing their SMART checks and no volumes mount. We recover by working our way up from the bottom through each layer instead of viewing the entire array as just one large unit.
Most common scenarios are: a RAID 5 loses a second drive during a rebuild; a pool becomes “Degraded” or “Inactive” due to loss of power in the middle of a write; a thin volume fills its pool and stops writing new data; and firmware updates cause the pool to become unmountable, although the array appears to remain in assembly. Snapshots also provide some useful complications: deleted files may exist in snapshots even though they’ve been removed from the active volume as long as the pool remains readable.
Every disk is pulled out of the NAS (with no attempt made to get any disks to rebuild) and labelled by bay. All disks are then imaged read-only. Where possible, weak areas on the disks will be retried in short passes instead of being hammered repeatedly. The mdadm set is then re-assembled using the images of all disks. This is achieved by detecting all relevant details of the mdadm set such as chunk size, member order, parity layout and data offset etc., rather than relying upon potentially stale superblocks. Thin-provisioning metadata within the LVM pool is then used to identify where the real extents lie and the file system is rebuilt from there. No request is ever made to the QNAP to assemble, repair or rebuild anything.
We need the internal disks removed if possible, and what matters most is bay order. So please label every disk with the bay number it was taken from prior to pulling it out of the NAS. Do not re-insert a disk that the NAS has ejected. Do not allow either a rebuild or resync process to run. And do not use the QNAP’s own repair tools to try to fix your problem first. If you already know which disk has failed, please let us know. It would save both parties diagnostic time. NAS work begins at £500 +VAT, however there is no charge for the initial 48 hour diagnostic period.
Usually yes. Often, each drive will have no signs of failure. Normally, when a problem occurs with the storage pool or thin provisioning layers above the RAID it causes the volume to disappear but the drives underneath remain intact. In other words, the rebuild process is done layer by layer using read-only images of each drive as they were at the time the failure occurred.
Each drive is captured in read-only format; then an attempt is made to reassemble the mdadm set below. This involves determining the chunk size, which disks were used (in what order), how parity was configured, and where data was placed based upon the structure of the original data, instead of using possibly outdated information contained within superblocks. Once this lower level has been successfully rebuilt, the LVM pool and thin volume are remapped over it and your file system is rebuilt from there; all of these processes occur outside of the device.
Often, yes. A snapshot maintains an older copy of the blocks a particular file occupies, therefore if a user deletes a file from the active volume, it usually remains accessible in one or more snapshots as long as the pool is still readable and you stop making changes to the NAS immediately after realising something has gone wrong.
Yes, if possible; we need the internal disks removed. Label each piece with its bay number first; bay order matters for reconstruction. Do not re-insert an ejected disk and do not let the unit run a rebuild or pool repair before it comes in.
NAS/RAID/server recovery costs from £500 +VAT, fixed within a written quote after a free diagnostic period of 48 hours. Where a member requires physical drive-level work, 50% deposit applies with balance only payable on successful recovery.