We have received preliminary root cause analysis from the building operator, Digital Realty. The system was throughly checked and is now operating normally at N+1 resiliency with no further risk to the load.
Full report is available to affected Customers.
We regret inconvenience caused by this incident.
Posted May 07, 2023 - 07:38 BST
Update
We are awaiting executive summary of the last night’s incident from Digital Realty to understand the cause of the power cut.
Updates will follow once they become available to us.
Posted May 06, 2023 - 16:44 BST
Monitoring
As per Digital Realty:
As an update to the UPS issue at 32 6th Ave, the site team in conjunction with the electrical vendor continues to investigate the source of the issue. UPS-A has been confirmed to be on-line at N+1 and stable, if you experience any further power issues related to the event please make contact. Further updates will be provided as additional information becomes available.
Posted May 06, 2023 - 07:20 BST
Identified
As per Digital Realty:
As an update to the UPS issue at 32 6th Ave, this event has been upgraded from a YELLOW to a RED due to multiple customer reports of impact. The site team in conjunction with the electrical vendor continue to investigate the source of the issue. Further updates will be provided as additional information becomes available.
Posted May 06, 2023 - 05:50 BST
Update
We have observed a power blip affecting a number of devices, across both feeds, in some of our racks. We are working with the building to understand the cause.
Some of our network devices at this location were affected by this disturbance and have rebooted. Some devices that were *not* affected affected appear to have triggered a bug causing the PFE of the router to not program correctly. This has caused MPLS black holing on these devices across some labels. This has most visibly affected the connection between some 32 Avenue of the Americas devices and one of our core routers in Secaucus, NJ where we pick up NYIIX causing loss for the traffic coming from this exchange. The devices have now been normalised and a vendor case has been opened to understand the cause of this bug.
We are continuing to chase Digital Realty to provide updates regarding the state of the power system at 32 Avenue of the Americas as well as the root cause for the momentary power loss issue.
We regret any inconvenience this incident might have caused you.
Posted May 06, 2023 - 05:49 BST
Investigating
As per Digital Realty:
YELLOW Event - UPS Issue / Reduced Redundancy 32 6th Ave New York, NY - 10th Floor
The engineering team at 32 6th Ave reports an issue with UPS supporting the 10th Floor resulting in a reduction of system redundancy. There has been no indication of interruption in critical IT load to equipment in the space. The site team has engaged additional resources to assist in investigation of the issue and subsequent repair. Further updates will be provided as additional information becomes available.
Posted May 06, 2023 - 02:57 BST
This incident affected: Datacentres (NYC1 - New York City, US).