Dear Dediserve Hong Kong users,
We'd like to apologize for the issues seen yesterday on HV2 in Hong Kong, the RFO for which is below.
Root Cause: Human Error
What happened?:
During routine disk replacement on one of our storage arrays the on-site engineer accidentally removed several incorrect disks from a degraded array, causing it to fall into a failed state.
What was done to correct this?:
All efforts were made to recover the array but unfortunately, this could not be done, so it was re-built and all services restored from the backups we took as a precaution before the work commenced.
What was the impact?:
All VMs were recovered successfully, however, any data transferred between the time of backup and restore will not have been retained.
What are we doing to prevent this in future?:
We are reviewing the incident with the DC and staff member involved and will be providing additional documentation and training to ensure this does not reoccur.
Apologies again for any inconvenience caused. Please get in touch if we can help any further.