UPS Failure

UPS failed while I was checking the batteries after the vendor finished battery maintenance. This was my last walk-through of the day. I checked the battery voltage and current at the UPS panel, inspected each battery for signs of leakage or post swelling. I was going to print a battery report when I noticed the battery monitor printer was near the end of the paper. I looked for a new roll of paper and just as I closed the door, the UPS failed releasing a fireball into the room. The failure tripped one of the three 3000 AMP main feeders, the battery breaker and the UPS input breaker. My first concern was all the smoke even though the fire sensors in this room are heat detecters. By the time I got the door open to vent the smoke and add light to the room, my phone was blowing up with alarms. Though I was quite busy for the next few hours, none of the computers in the center were affected.

Category:

Science & Technology

Tags:

• ups explosion
• plasma
• tripped breaker
• fire ball
• explosion
• disaster
• recovery
• data
• center
• arc flash
• electric arc

[TheDennisChannel]

Not according to the hardware logs from the UPS.

[northerbrewer]

Did anyone ever consider ARC fault risks?



[TheDennisChannel]

@northerbrewer

Yes, when work is being done on high energy circuits. People walk through the space frequently, though. You can't suit up for everything done in the space.

[northerbrewer]

@TheDennisChannel

I think it would be wise to restrict access to this area.

People should not " walk through frequently. "

Considering arc fault risk only when work is being done is rather foolish since all the equipment in that vault apears to be open and vented rather than designed to contain a flash.

You don;t need to suit up for every situation, there are various levels of protection for different levels of risk,

I am typing this today today because I survived a flash

[TheDennisChannel]

@northerbrewer

Security guards walk through on a regular schedule. The area is quite restricted, with biometric readers and only a few people have access to the spaces.

I trust you are O.K. after your arc flash experience.

[oconnellcom]

is that romex hanging down in the bottom left corner of the video?

[TheDennisChannel]

@oconnellcom

No, ROMEX is not used in data center. Circuits are individual wires in conduit. What you see is a nylon pull string that was left in an empty conduit. The pull string can be used later to pull wire through a conduit when there is a need for circuit changes in the future. The nylon string blows in the beeze caused by the air conditioners. It's harmless except it could trigger the video recorder to record all the time if it was a large percentage of the camera's field of view.

[evilunixuser1]

At my old company, the power generator failed, spilling diesel fuel into the Data Center, top THAT.

[TheDennisChannel]

@evilunixuser1

You win!

[meggerbiddle]

Maybe IGBT failure in the inverter section? Big bang from this, full force of the batteries behind them. I doubt any non-engineer would be allowed to open the cubicle door to that section without full isolation. Probably interlocked anyway?

[produKtNZ]

So this was a failure as a result of careless battery maintenance performed by whichever company you contracted to do the maintenance?

[daltxn]

@northerbrewer And by the way, I'm glad you're well after your arc flash incident.

[daltxn]

@northerbrewer UPS modules contain fans for internal cooling. The flash is seen through in the air intake located at the base of the unit and at the top where air is exhausted. There are doors (closed in this video) on the front of the unit which only allow access to control switches and additional covers behind the doors covering live parts to contain a flash. The last part of the video shows those covers removed. Just looking at this unit I'd guess it's a Liebert brand UPS.

[DOSeater]

And there you have it guys. Never grow mustaches

[northerbrewer]

@TheDennisChannel

I am fine just a little hearing loss.

Maybe put something up to keep people away from the enclosers. I am not familiar with American codes but I would think a 3-4 foot exclusion area would prevent someone from catching fire.

Arc fault level 0 PPE prevented me from being significantly burned, I am very happy to about that....

I remember all the live work, Silpacs and open buss I used to do many years ago and consider myself lucky they never flashed on me.