IT incident 5 May 2017 – a response from ISD and Estates
16 May 2017
We wanted to update UCL staff and students following the major loss of IT services on Friday 5 May 2017. Firstly, we want to apologise for the significant impact this had on staff and students’ ability to work across the whole university. We all rely on this technology round the clock and any loss of service is frustrating.
The incident occurred when a contractor working in a plant room near to one of our datacentres damaged one of the main cooling pipes that feed the datacentre and this led to a failure of the cooling system. Very quickly the temperatures in the datacentre started climbing and temperature sensors set off alarms which both ISD and Estates staff responded to immediately. In a race against time, technical teams attempted to migrate services over to our offsite data centre before the high temperatures caused the systems to automatically shutdown to avoid damaging the hardware. In many cases the team managed this without interrupting services but in some cases services were lost for a short period. Within a couple of hours almost all services were back up and running. Estates were able to repair the damaged pipe quickly but then it took several hours to refill the cooling system. Normal temperatures and operation of the datacentre was restored around 7pm. There was no data loss or loss of data integrity as a consequence of this emergency datacentre shutdown. ISD operates with full resiliency and backups for all primary systems to minimise disruptions when events such as this occur, however in emergency situations like this there can be a short outage to some services for which we apologise.
ISD and Estates teams responded to the incident, however such incidents are a reminder for all departments to consider operational back up of data systems and their own Business Continuity plans.
A key factor in this case was the large amount of construction and related work taking place on and around the Bloomsbury campus at the moment and therefore operating procedures have been changed to address the increased risk.
ISD and Estates will continue to work closely together to mitigate risks while this work is on-going.
Once again please accept our apologies for this disruption to services.
Estates and Information Services Division