Utility Power Outage / UPS1 Failed to Bypass

Incident Report for Joe's Datacenter, LLC

Postmortem

We apologize for the inconvenience, but we experienced a power outage from the utility and had to run on generator power. One of the two UPS systems had an issue that caused an outage to some of the clients in the facility. We were able to get power back and have gone rack to rack making sure as much as possible has been powered back on. If you are still experiencing an issue please respond to this ticket and a technician will investigate further. Otherwise this ticket is being closed as we believe this issue has been resolved. We apologize for the inconvenience this caused, we are reviewing this internally to see what better can be done to prevent this in the future.On October 28th there was a loss of utility power causing the data center to run on UPS and Generator power. Problems with the Generator and transfer switch configuration caused it to shut down and had to be manually restarted several times before becoming fully operational. During that short window, one of the two UPS systems ran out of backup battery and switched into bypass mode, which was unavailable due to the outage. In order to prevent damage to the equipment connected to that UPS, the breakers were manually opened (shut off) until the UPS system could be brought back online. The other UPS was able to stay online and the customers on that equipment did not lose power. Data center HVAC equipment was temporarily down, however it came back online shortly after the generator was fully started. Temperatures in the data-center stayed within acceptable limits the entire time. After the utility power was restored, we systematically walked the data center, powering on equipment that did not automatically turn on. We then worked with clients on a one on one basis via the ticket system and phones to make sure everyone was back on-line as quickly as possible. Additional staff was called in and the majority of customers were offline for 30 minutes to 1 hour. Some customers had other hardware and software issues that had to be resolved, which resulted in additional time without service.

Posted Nov 06, 2019 - 15:15 CST

Resolved

All Data Center critical infrastructure is back operational. We will follow up with individual clients to resolve issues related to the outage.

Posted Oct 28, 2019 - 07:10 CDT

Monitoring

Utility Power has been restored and all circuit breakers have been turned back on. We are going cabinet to cabinet and make sure customer equipment is powered back on.

Posted Oct 28, 2019 - 06:56 CDT

Identified

The backup generator shutdown due to an over-voltage error. Manually restarting the generator was only successful after several manual attempts. In the process UPS1 switched into bypass mode, to prevent damaging equipment circuit breakers have been manually shut off to equipment running on UPS1. This issue has been elevated to a partial outage. Some Colocation and Dedicated Servers are without power. UPS2 is still operational on battery with generator backup power.

Posted Oct 28, 2019 - 06:25 CDT

Update

We have contacted the utility company and were notified they are aware of an issue at the sub-station and they have already dispatched technicians. They believe power will be restored with a simple breaker flip at the substation.

Posted Oct 28, 2019 - 06:20 CDT

Investigating

There has been a loss of Utility Power. The transfer switch is starting the Generator and we are currently operational running on UPS battery backup. UPS systems have an estimated 20 Min of battery run-time.

Posted Oct 28, 2019 - 06:11 CDT

This incident affected: Power Infrastructure.