Behind the Headlines: Super Bowl Blackout
Independent third party being hired to investigate partial power outage during Super Bowl
- by Linda Deckard
- Published: February 6, 2013
A power outage causes a 34-minute game delay at Super Bowl XLVII in at the Mercedes-Benz Superdome in New Orleans.
Three days after a partial power outage delayed play during Super Bowl XLVII Feb. 3 at the Mercedes-Benz Superdome in New Orleans, the root cause is still undetermined.
Doug Thornton, SMG regional VP on site, is leading the follow-up investigation for the management firm. The most recent official statement issued Feb. 5 comes from Entergy New Orleans Inc., the Louisiana Stadium and Exposition District (LSED) and SMG, stating that those parties are selecting an independent third party to assess the cause of the outage. Entergy is also continuing its own root-cause analysis.
Until the reports are in, the fact of the matter is no one knows what caused the power outage, which delayed play for 34 minutes at the beginning of the second half, which had the Baltimore Ravens trouncing the San Francisco 49ers in the National Football League’s premiere championship game.
Alan Freeman, general manager of the venue for SMG, confirmed, “We know where it occurred, but not why it occurred. It happened at the intersection of the utility company’s power to us and us hooking into the power.”
Both Thornton and Freeman confirmed they had upgraded the power system several months ago. After power outages in the last year or so at Candlestick Park, San Francisco, and MetLife Stadium, East Rutherford, N.J., SMG decided to run some tests and discovered some slight erosion of their capacity at the switch box.
“The last thing we want was any problems during the Super Bowl, so we decided to upgrade those two feeder lines,” Freeman said. Each line feeds 50 percent of the building, east and west.
On Sunday, one of those lines went down, leaving the dome with 50 percent of lighting. Fortunately, the upgrade made it possible to switch the load to one line if one of the two went down.
“We were about to do it, but the utility company called us at the last minute and said don’t do anything, we are going to restore the power to the line that’s down and we’ll be back up to full speed before you know it,” Freeman said. “ And that’s what happened.”
The partial power outage that delayed Super Bowl at the Mercedes-Benz Superdome has got a lot of attention, but cannot change the fact that the event was highly successful. Concessions and catering, managed by Centerplate, grossed more than $5 million, phenomenal numbers. The National Football League reported ticketed attendance of 71,024. That does not include press and staff. “We had 5,000 workers here,” said Thornton.
But the aftermath has been all about the outage. “The last couple of days have been very long,” Thornton said. “To me the ultimate irony is I was sitting in my office at 2:30 a.m. on Monday morning after dealing with all the firefighting, and I realized that was the same conference table I was sitting at the night before I evacuated out after [Hurricane] Katrina.”
Thornton knows about being prepared, and the Superdome staff was good to go.
“Several weeks ago we simulated a power outage so we could document all of the conditions and spaces that are on the emergency power grid and circuit breakers that had to be turned off or opened and what needed to be restarted, and the sequence. We know those things, but we wanted to be certain.”
They documented where to position people around the building to restart all of the systems, including arena lights, scoreboards, telephones, telecommunications, and network servers. “We prepared for it, drilled for it and wrote a protocol for the sequencing process. The voice you heard on the PA [during the outage] was a script we and the NFL had prepared for that particular circumstance,” Thornton said.
The question now is not how it was handled, but how it was caused. SMG, LSED and Entergy will hire a third party to determine the cause by the end of this week. It will take two to three weeks, best-case scenario, to get the report. “We’re reasonably confident the Superdome infrastructure and power distribution systems are not related to this power outage because when Entergy restored service to power the switch gear, everything was powered up again like it never shut down. No blown fuses, transformers, busted circuits. We have a strong belief the infrastructure is not the root cause,” Thornton said.
He also noted that the loads required for Super Bowl, which brings in auxiliary generators for the half time show and CBS, were 66 percent of capacity. “We had a lot of end room. It was not an overloading situation.” The load levels were consistent with any New Orleans Saints football game or the Sugar Bowl.
“As part of analysis we will reconstruct incremental power, but it was minimal compared to overall usage here,” he said.
And there were no abnormalities in the metering. Freeman said they were monitoring load levels every five to ten minutes during the first half and every two minutes during the halftime show. No surges or spikes were noted. And the lights powered up again without a problem after half time.
“There were no blown fuses, faults or shorts in the building, but we’re not taking that for granted. We’re doing the study,” Thornton said.
Four firms presented regarding the forensic electrical engineering analysis, Thornton said. The selection will be mutually made by SMG, LSED and Entergy.
Why investigate? “To fix the problem and assure ourselves it will never happen again,” Thornton said.
THE SEQUENCE OF EVENTS
When all is said, it was an inconvenience, not a life-threatening situation. In the life of a stadium manager, it was not the worst thing that can happen. The emergency power worked, the emergency plan worked, the staff dealt with the public, power was restored quickly.
“But we need to restore total confidence in the entire power system here, not just the dome,” Thornton added.
He has already talked to executives at MetLife Stadium, and the similarities are numerous. “Their configuration of power supply is similar to ours, with A and B feeds that take 13,800 volts from their power company through offsite switch gear, just like we do. The device that disconnected our power is owned by Entergy Corp. The switch gear senses various abnormalities that could cause the system to shut down.”
At MetLife, they found an improperly set relay or fault sensing device, which is human error. “As I understand it, theirs was a power company problem,” Thornton said.
When the partial power outage occurred, Thornton was in the NFL Control booth on the press level. “60 Minutes” was filming a documentary about the Super Bowl and was there with them. The show was to air tonight.
“The half time show just concluded. We brought the house lights back up and were running the fans to clear the pyro smoke and restoring play. The kickoff was returned. Within two minutes of the restart of play, the lights in our booth went off. We had physically turned off those lights during the halftime show. We thought someone had accidentally hit the switch again.
“As I turned to look, I saw in the corner of my eye that the west side stadium lighting was starting to go through its sequencing of clicking off. My heart stopped. In my mind, I knew that meant we had a loss of power in our A feeder.”
He was immediately on the radio with the engineers who confirmed the power problem. The protocol to transfer power from the A leg to the B leg commenced. They coordinated everything with Entergy personnel, located 1,280 feet from the building, behind the dome. “Just as we were ready to transfer it over, they said hold off because we can restore power to the A leg. They did.”
When cold, it takes 9 minutes for the 600-foot candles to come back up, he said. Thus the delay in total was 34 minutes. The west side never shut down, including concessions.
“Be prepared,” Thornton said of his best advice post-Super Bowl. “I’m very pleased with the way our team operated under pressure. SMG’s team and Entergy’s team remained calm and worked through the problem. It was remarkable to get back up in 23 minutes.”
Up to that power outage, and the whole week prior, New Orleans had rolled out the red carpet. The Superdome had no issues getting people into the building. The average wait time at magnetometers was no more than 10 minutes,” Thornton said.
Traffic was redirected through foot soldiers on the ground and through text messaging capabilities. In fact, ticket buyers received text messages to return to their seats after the power was revved back up.
“Even though it was an uncomfortable situation, very uncomfortable, looking back, the preparation, the work done on the front end by us and the NFL paid off. We were able to restart play. It was an inconvenience, but it was not catastrophic,” Thornton said.
Interviewed for this story: Doug Thornton, (504) 587-3827; Alan Freeman, (504) 587-3892
- by Linda Deckard
- Published: February 6, 2013