Loss of services at 11am on 5th February

alert

[05 February 2016]

All services should now be back following the outage earlier today.  While the underlying problem was fixed quite quickly the recovery of individual services has taken some time. This also explains why some systems were back before others.

Just before 11am today, 5th February, a problem was identified during a standard network configuration change. The problem affected the connections between data centre servers and the storage service. The network was restored at 11.20am, and we then began restoring the underpinning infrastructure services (storage, databases, etc)

Communications were hampered by the outage to both the website and mailing list service. Emails were finally distributed through an alternative route using Goggle and both Yammer and the IT Services' Facebook page were used to update staff and students.

The majority of services were back by 1pm, and almost all back by 1.30pm. At the time of writing the most significant system still experiencing problems is the Plone web CMS (the platform is used for intranets). This is having knock-on effects on some other webapps with a dependency on Plone.

We will continue to monitor services to ensure a full recovery. If you do have any issues please contact the IT Services Desk.

IT Services apologises for the the problems today and thanks you for your patience as we got services back.