Both feeds A and B are fully operational with full UPS capacity on feed 'B' and half of the UPS capacity on feed 'A'.
We do not consider the site at risk any more.
Any remedial works will be communicated separately.
Posted Feb 21, 2023 - 20:35 GMT
Update
Vendor update: Equinix IBX Site Staff reports that the vendor has identified and removed a faulty battery from UPS A1 system. UPS A1 will remain isolated while further investigation works are carried out. All customer loads remain stable on UPS A2 at reduced redundancy.
* Situation Status: Stable * Equipment/Condition: UPSS - LD8.8-1200KVA UPS Module - UPS A1 (B8 - UPS System 4), LD8.8-1200KVA UPS Module - UPS B2 (B8 - UPS System 4), LD8.8-1200KVA UPS Module - UPS B1 (B8 - UPS System 4), LD8.8-1200KVA UPS Module - UPS A2 (B8 - UPS System 4) | Out of service * Current System Redundancy: Reduced Redundancy * Resources on Site/Pending: Vendor Arrived * Generator Fuel Run Time (hrs.): N/A * Current Colo Temperature Status: N/A * Current Colo Humidity RH Status: N/A
The next update will be sent when there is a significant change to the situation.
Posted Feb 21, 2023 - 16:31 GMT
Update
Vendor update: Equinix IBX Site Staff reports that the vendors have arrived onsite to carry out further investigation works on UPS A1 and its battery system. UPS A1 will be isolated by the vendor and all customer loads will be supported by UPS A2 at reduced redundancy.
* Situation Status: Stable * Equipment/Condition: Battery Strings for UPSs A2, B1, and B2 Reinstated * Current System Redundancy: Reduced Redundancy * Resources on Site/Pending: Vendor Arrived * Generator Fuel Run Time (hrs.): N/A * Current Colo Temperature Status: N/A * Current Colo Humidity RH Status: N/A
The next update will be sent when there is a significant change to the situation.
Posted Feb 21, 2023 - 12:38 GMT
Investigating
We have received the following update from the vendor:
Equinix IBX Site Staff reports that battery strings for UPSs A2, B1, and B2 have been reinstated. IBX Critical Facility Engineers will isolate UPS A1 to ensure the load is supported on UPS A2. The specialist UPS vendor will attend the site to identify the root cause of the issue on UPS A1 and perform remedial works. The ETA for the specialist UPS vendor is 3 hours. All customer equipment is now UPS backed.
* Situation Status: Stable * Equipment/Condition: Battery Strings for UPSs A2, B1, and B2 Reinstated * Current System Redundancy: Reduced Redundancy * Resources on Site/Pending: Vendor ETA on site 3 hrs. * Generator Fuel Run Time (hrs.): N/A * Current Colo Temperature Status: N/A * Current Colo Humidity RH Status: N/A
The next update will be sent when there is a significant change to the situation.
---
Clouvider continues to consider the site at risk.
Posted Feb 21, 2023 - 11:14 GMT
Identified
We have received the following vendor update:
Equinix IBX Site Staff reports that IBX Critical Facility Engineers have identified an issue with UPS string A1. Strings A2, B1 and B2 are being reinstated shortly. Prior to reinstating the UPS strings, safety checks will be carried out. The expected time for UPS strings A2, B1 and B2 to come online is approximately 20 minutes if no further issues are discovered.
* Situation Status: Stable * Equipment/Condition: Isolated batteries for UPSs A1, A2, B1, and B2 within Buildings 8 and 9 * Current System Redundancy: Reduced Redundancy * Resources on Site/Pending: Vendor Arrived * Generator Fuel Run Time (hrs.): N/A * Current Colo Temperature Status: N/A * Current Colo Humidity RH Status: N/A
The next update will be sent when there is a significant change to the situation.
Posted Feb 21, 2023 - 10:16 GMT
Investigating
We have been informed of the following incident at Equinix LD8 in London.
Equinix LD8 is one of our two core routing sites for London as well as a location of service for some Dedicated Server Customers. The London network is therefore at risk for all Customers. Furthermore, Customers with servers in London and server ID starting with a letter S are at additional risk.
Vendor notification:
Equinix IBX Site Staff reports that due to a fire discovered within the UPS battery room within Building 8, and 9, UPSs A1, A2, B1, and B2 batteries have been isolated. The customer loads remain dual-fed on utility power with no battery backup available. The fire brigade is further investigating the issue.
* Situation Status: Under Investigation * Equipment/Condition: Isolated batteries for UPSs A1, A2, B1, and B2 within Buildings 8 and 9 * Current System Redundancy: Reduced Redundancy * Resources on Site/Pending: Vendor Contacted * Generator Fuel Run Time (hrs.): N/A * Current Colo Temperature Status: N/A * Current Colo Humidity RH Status: N/A
Clouvider is monitoring the situation.
Posted Feb 21, 2023 - 10:04 GMT
This incident affected: Customer Portal (Customer Portal, Service Management API, OnApp Management Interface, Legacy Customer Portal), Network (MPLS/VPLS Services, London, UK Core Network, Upstreams), Hosted Services (Anycast DNS, Web Hosting, Public Cloud, Private Cloud, CDN, Dedicated Servers), and Datacentres (Harbour Exchange - London, UK).