Hong Kong HV2 - RFO

Scheduled on 21/02/2019 13:30:00 Status Resolved Fault / Issue Estimated finish 21/02/2019 18:00:00

Dear Dediserve Hong Kong users,

We'd like to apologize for the issues seen yesterday on HV2 in Hong Kong, the RFO for which is below.

Root Cause: Human Error

What happened?:
During routine disk replacement on one of our storage arrays the on-site engineer accidentally removed several incorrect disks from a degraded array, causing it to fall into a failed state.

What was done to correct this?:
All efforts were made to recover the array but unfortunately, this could not be done, so it was re-built and all services restored from the backups we took as a precaution before the work commenced.

What was the impact?:
All VMs were recovered successfully, however, any data transferred between the time of backup and restore will not have been retained.

What are we doing to prevent this in future?:
We are reviewing the incident with the DC and staff member involved and will be providing additional documentation and training to ensure this does not reoccur.

Apologies again for any inconvenience caused. Please get in touch if we can help any further.

Related servers / services

RFO - London 3 fault

Scheduled on 26/01/2019 12:00:00 Status Resolved Fault / Issue Estimated finish 26/01/2019 15:00:00

Dear Dediserve London users,

You may have noticed service being impacted on Saturday 26th of January, for which we sincerely apologize. Below is the formal RFO for the incident, please rest assured we are taking all reasonable measures to make sure this does not re-occur, thank you for your understanding.

What was the cause?:
A faulty PDU caused damage to core switching, interrupting connectivity to VMs routed through that stack.

What was the fix?:
The faulty PDU and core switch stack had to be replaced, replacing the PDU and re-configuring the switch stack was time-intensive and contributed to the length of downtime.

What was the impact?:
A notable number of VMs lost connectivity whilst the switch stack and PDU was replaced, a number also required reboot to clear their ARP cache to restore connectivity following the hardware change.

Will the issue re-occur?:
There should be no reason this issue would re-occur, we are performing staggered upgrades to power equipment in this site to better protect against any future occurrence.

As always, if you have any questions, please let us know.

Related servers / services

Hong Kong Cloud DC Migration

Scheduled on 26/01/2019 23:00:00 Status Finished Estimated finish 27/01/2019 03:00:00

Dear Dediserve users,

Please be aware that we will be performing a Datacenter migration on the 26th of January between 11pm and 3am (HKT) of the 27th, we are migrating our existing Hong Kong cloud to a higher-tier DC in the interest of providing better, more reliable service for you, the end-user.

You do not need to do anything, we will power down all services, migrate the cloud and power everything back up in line with our schedule - all IPs will remain the same and service should be restored within the 4-hour window allocated.

As always, if you have any questions please let us know.

Related servers / services