Maybe IGBT failure in the inverter section? Big bang from this, full force of the batteries behind them. I doubt any non-engineer would be allowed to open the cubicle door to that section without full isolation. Probably interlocked anyway?
Yes, when work is being done on high energy circuits. People walk through the space frequently, though. You can't suit up for everything done in the space.
I think it would be wise to restrict access to this area.
People should not " walk through frequently. "
Considering arc fault risk only when work is being done is rather foolish since all the equipment in that vault apears to be open and vented rather than designed to contain a flash.
You don;t need to suit up for every situation, there are various levels of protection for different levels of risk,
I am typing this today today because I survived a flash
Security guards walk through on a regular schedule. The area is quite restricted, with biometric readers and only a few people have access to the spaces.
I trust you are O.K. after your arc flash experience.
Maybe put something up to keep people away from the enclosers. I am not familiar with American codes but I would think a 3-4 foot exclusion area would prevent someone from catching fire.
Arc fault level 0 PPE prevented me from being significantly burned, I am very happy to about that....
I remember all the live work, Silpacs and open buss I used to do many years ago and consider myself lucky they never flashed on me.
@northerbrewer UPS modules contain fans for internal cooling. The flash is seen through in the air intake located at the base of the unit and at the top where air is exhausted. There are doors (closed in this video) on the front of the unit which only allow access to control switches and additional covers behind the doors covering live parts to contain a flash. The last part of the video shows those covers removed. Just looking at this unit I'd guess it's a Liebert brand UPS.
No, ROMEX is not used in data center. Circuits are individual wires in conduit. What you see is a nylon pull string that was left in an empty conduit. The pull string can be used later to pull wire through a conduit when there is a need for circuit changes in the future. The nylon string blows in the beeze caused by the air conditioners. It's harmless except it could trigger the video recorder to record all the time if it was a large percentage of the camera's field of view.
This configuration is commonly known as a "catcher buss". The computer loads are connected to 2 PDUs from 2 different UPS, utility transformers and/or generators. The PDUs have Automatic Static Transfer Switches that move the computer load to a reserve UPS should a source fail. This happens automatically or it can be done manually for maintenance.
In this event, it was the reserve UPS unit that failed. The reserve UPS carries no load under normal conditions.
The space is ventilated by a rooftop package air conditioner that pressurizes the room and purges any gases from the batteries through vents in the roof. When a fire is detected by the fire alarm control panel, a fire damper closes to remove outside air. In this case, there was no fire detected since the heat detecters in the room didn't get hot enough. The fireball was contained by the UPS enclosure and suppressed by removal of the energy source when the breakers tripped.
It is a Liebert Series 600-750 KVA (675 kW) which was installed in 2000. Service was fine. Liebert rebuilt the module on site and it has run for two years. I have installed another one since then.
@evilunixuser1: Back in 2008 the power bus bars at Fisher Plaza in Seattle exploded. As this is a mixed-use building, city fire code requires wet and dry suppression systems in this area. Power bars blew, setting off the sprinkler systems. The sprinkler systems blew out the generators as they attempted to come online. Then they spilled diesel fuel into the building. Building ran off truck-based gensets for several months. This was an Internap Colo.
Leave it up to the geniuses in IT to install a fire detector that only detects heat and not smoke! IT is the biggest problem in any and every organization.
oof, if that happened where I used to work then all alarmbells would go off immediately. We had a VESDA line running over all components in the power-room.
It may be an odd question, but is it normal for a building to have the batteries in the same room as the UPS? I mean, in case of a fire in either, things can go from bad to worse pretty quick.
VESDA would be OK if it was attached to the Sprinkler Pre-Action Water Valve. In this case the valve would open but no water would flow until a sprinkler head heated to the point where it opened. There is a potential for smoke (and other gases) in UPS rooms so very early detection runs the risk of being too sensitive and being a nuisance. Heat detectors seem to be the best solution.
It's common for VRLA batteries to be in the same room as the UPS.
Had a 500kVA UPS burn out a filter circuit while I was in the room with the techs doing maintenance, blew about 6 or 7 diodes on a filter module card, cracked most of them and one basically evaporated. That was fun. Bit of smoke produced. Thankfully no VESDA in that room to react. I isolated it anyway, just to be on the safe side.
They replaced the module, the unit was put back in service. Thankfully we were N+1 and nothing else was affected.
Semiconductors (diodes, SCRs, IGBTs, transistors, etc) have one rather undesirable characteristic: when they fail, they like to go short.
I don't deal with equipment anywhere near this power level, but have seen my fair share of spectacular catastrophic failures. One of our 15kVA 240V single phase UPS failed in a similar way. A toy compared to the UPS in your video, but I'll tell you what, that little UPS failed in a rather spectacular way. Lots of arcs and smoke.
@TehMG I work at a company that manufactures UPS's, rectifiers and power conditioners. I know about shorted semiconductors-They go POW or BOOM! Especially 380 volts and higher they blow violently. Our stuff has fuse links and fuses/breakers but those can't react fast enough to protect an IGBT or SCR that gets switched on at the wrong time or against another device in the circuit failing on it's own. Tap switchers and inverters especially since the devices are in parallel.
@TehMG With semiconductors it only takes microseconds for a junction to melt and short the layers together. That was an advantage of tubes, a momentary overload or overvoltage didn't make them fail and if they did fail it was usually a burned out heater or loss of emission. Shorted failures were rare with tubes and about the only way they could short was from mechanical shock or extreme overloading and that took time. Cathodes, grids and plates never touch unlike PN junctions.
I learned that the failure in the UPS was a phase-to-phase short on the input 480 VAC circuit, apparently through the diodes or SCRs in the input AC side. Once the plasma ball happened, the current increased so quickly it amounted to a "bolted fault". I'm told by the electrical engineer of record that breakers don't always coordinate on a bolted fault. I'm happy the one that tripped kept it in that branch circuit rather than going up to the 22kV utility source.
@jimbomassoud The sequence alternator in the buffer setting module may need to be reset to compensating mode. Otherwise, your buffer joint and vertical transformer may overload the interface module in the primary buffer setting of the joint processor in the main shaft ring of your third sequencer.
Make sure you update your third shaft ring sequencer as well by uploading the current firmware.
@backinthecrystal I believe that a Allen-Bradley retro entabulator should have been considered for installation between the vertical transformer & the buffer joint. For more information, you can view the Allen-Bradley retro entabulator through a YouTube search.
@gblueslover2 But a retro entabulator will not get rid of signal degradation between the charlie joints of the post-linear compensator in the internal buffeting jacket. The buffer joint must be connected to a ventricular pump module that is ramscooped into a redundant kiloquad buffer so that the alignment is no less than three millimeters between the signal processor joint and key buffer jacket ring.
Its all good till you let the smoke out. Remember, you must keep smoke (and fire) in at all times. You're lucky you were looking at the battery monitoring system at that point and not with your face in the UPS looking for the serial number to give to tech support.
Maybe IGBT failure in the inverter section? Big bang from this, full force of the batteries behind them. I doubt any non-engineer would be allowed to open the cubicle door to that section without full isolation. Probably interlocked anyway?
meggerbiddle 1 month ago
Not according to the hardware logs from the UPS.
TheDennisChannel 1 month ago
So this was a failure as a result of careless battery maintenance performed by whichever company you contracted to do the maintenance?
produKtNZ 1 month ago
And there you have it guys. Never grow mustaches
DOSeater 1 month ago
Did anyone ever consider ARC fault risks?
northerbrewer 2 months ago
@northerbrewer
Yes, when work is being done on high energy circuits. People walk through the space frequently, though. You can't suit up for everything done in the space.
TheDennisChannel 2 months ago
@TheDennisChannel
I think it would be wise to restrict access to this area.
People should not " walk through frequently. "
Considering arc fault risk only when work is being done is rather foolish since all the equipment in that vault apears to be open and vented rather than designed to contain a flash.
You don;t need to suit up for every situation, there are various levels of protection for different levels of risk,
I am typing this today today because I survived a flash
northerbrewer 2 months ago
@northerbrewer
Security guards walk through on a regular schedule. The area is quite restricted, with biometric readers and only a few people have access to the spaces.
I trust you are O.K. after your arc flash experience.
TheDennisChannel 2 months ago
@TheDennisChannel
I am fine just a little hearing loss.
Maybe put something up to keep people away from the enclosers. I am not familiar with American codes but I would think a 3-4 foot exclusion area would prevent someone from catching fire.
Arc fault level 0 PPE prevented me from being significantly burned, I am very happy to about that....
I remember all the live work, Silpacs and open buss I used to do many years ago and consider myself lucky they never flashed on me.
northerbrewer 2 months ago
@northerbrewer UPS modules contain fans for internal cooling. The flash is seen through in the air intake located at the base of the unit and at the top where air is exhausted. There are doors (closed in this video) on the front of the unit which only allow access to control switches and additional covers behind the doors covering live parts to contain a flash. The last part of the video shows those covers removed. Just looking at this unit I'd guess it's a Liebert brand UPS.
daltxn 1 month ago
@northerbrewer And by the way, I'm glad you're well after your arc flash incident.
daltxn 1 month ago
new pair of underpants needed.
burn19ballz 2 months ago
is that romex hanging down in the bottom left corner of the video?
oconnellcom 3 months ago
@oconnellcom
No, ROMEX is not used in data center. Circuits are individual wires in conduit. What you see is a nylon pull string that was left in an empty conduit. The pull string can be used later to pull wire through a conduit when there is a need for circuit changes in the future. The nylon string blows in the beeze caused by the air conditioners. It's harmless except it could trigger the video recorder to record all the time if it was a large percentage of the camera's field of view.
TheDennisChannel 3 months ago
Hi Denis was this 4unit configured in parrallel?How was it not affect on the others unless there are not Neutral link:)tx
monkeycups1 3 months ago
@monkeycups1
This configuration is commonly known as a "catcher buss". The computer loads are connected to 2 PDUs from 2 different UPS, utility transformers and/or generators. The PDUs have Automatic Static Transfer Switches that move the computer load to a reserve UPS should a source fail. This happens automatically or it can be done manually for maintenance.
In this event, it was the reserve UPS unit that failed. The reserve UPS carries no load under normal conditions.
TheDennisChannel 3 months ago
Good ventillation but in case of fire maybe it would be a good idea to turn it off.
guzguz21 3 months ago
@guzguz21
The space is ventilated by a rooftop package air conditioner that pressurizes the room and purges any gases from the batteries through vents in the roof. When a fire is detected by the fire alarm control panel, a fire damper closes to remove outside air. In this case, there was no fire detected since the heat detecters in the room didn't get hot enough. The fireball was contained by the UPS enclosure and suppressed by removal of the energy source when the breakers tripped.
TheDennisChannel 3 months ago
CRIKEY!
interplexer 3 months ago
Atleast you found the fault.
beaman220 4 months ago
No sir, your package arived unharmed, and on time.
policedylan 4 months ago
nearly electrocuted !!! We lost a wonderful ogrish scene.
mptrax 4 months ago
must have been a emerson 610 unit. Have had a couple of those units blow at the DC i work at.
nweishuhn 4 months ago
@nweishuhn
It is a Liebert Series 600-750 KVA (675 kW) which was installed in 2000. Service was fine. Liebert rebuilt the module on site and it has run for two years. I have installed another one since then.
TheDennisChannel 4 months ago
At my old company, the power generator failed, spilling diesel fuel into the Data Center, top THAT.
evilunixuser1 4 months ago 28
@evilunixuser1
You win!
TheDennisChannel 4 months ago 16
@evilunixuser1: Back in 2008 the power bus bars at Fisher Plaza in Seattle exploded. As this is a mixed-use building, city fire code requires wet and dry suppression systems in this area. Power bars blew, setting off the sprinkler systems. The sprinkler systems blew out the generators as they attempted to come online. Then they spilled diesel fuel into the building. Building ran off truck-based gensets for several months. This was an Internap Colo.
xavoc 4 months ago
"what does this button do?"
"OH SHIT!!!"
LOL
clayto1332 4 months ago 2
most bat cabs are located in the same room if not right next to the ups next and clcs
dedflesh 5 months ago
he opened a portal to the future and came back every now and then!!!
SlugSwaggaObi 5 months ago
Leave it up to the geniuses in IT to install a fire detector that only detects heat and not smoke! IT is the biggest problem in any and every organization.
PilotVBall 5 months ago
@PilotVBall not IT, morons. Morons are biggest problem in any and every organization.
666Tomato666 4 months ago
I wonder if there were a UPS to the UPS..
dtiydr 5 months ago 3
@dtiydr
This location has four, so no computers were affected. It takes three or less to run the site.
TheDennisChannel 5 months ago
@TheDennisChannel It was a joke not a question. :)
dtiydr 5 months ago
oof, if that happened where I used to work then all alarmbells would go off immediately. We had a VESDA line running over all components in the power-room.
It may be an odd question, but is it normal for a building to have the batteries in the same room as the UPS? I mean, in case of a fire in either, things can go from bad to worse pretty quick.
BarneySaysHi 6 months ago
@BarneySaysHi
VESDA would be OK if it was attached to the Sprinkler Pre-Action Water Valve. In this case the valve would open but no water would flow until a sprinkler head heated to the point where it opened. There is a potential for smoke (and other gases) in UPS rooms so very early detection runs the risk of being too sensitive and being a nuisance. Heat detectors seem to be the best solution.
It's common for VRLA batteries to be in the same room as the UPS.
TheDennisChannel 6 months ago
Are you sure there was no mouse inside the UPS?
Dongskie2 7 months ago
@Dongskie2 Depends, was it a PS/2, serial mouse or USB?
Fraggle24 7 months ago
glad you weren't hurt there shipmatey
lexichronicle2 7 months ago
I'm assuming you literally Crapped your pants.
bait28 7 months ago
Holy shit this is the smartest group of comments on all of youtube. Sorry to mess it up, but wow!
FoxHoundUnit89 7 months ago
What was the make and model of this UPS?? Assuming this is in the states?
LouiLongcat 7 months ago
@LouiLongcat I'd like to know as well. Dennis?
karbonphiber 6 months ago
Had a 500kVA UPS burn out a filter circuit while I was in the room with the techs doing maintenance, blew about 6 or 7 diodes on a filter module card, cracked most of them and one basically evaporated. That was fun. Bit of smoke produced. Thankfully no VESDA in that room to react. I isolated it anyway, just to be on the safe side.
They replaced the module, the unit was put back in service. Thankfully we were N+1 and nothing else was affected.
shades2 8 months ago
Semiconductors (diodes, SCRs, IGBTs, transistors, etc) have one rather undesirable characteristic: when they fail, they like to go short.
I don't deal with equipment anywhere near this power level, but have seen my fair share of spectacular catastrophic failures. One of our 15kVA 240V single phase UPS failed in a similar way. A toy compared to the UPS in your video, but I'll tell you what, that little UPS failed in a rather spectacular way. Lots of arcs and smoke.
TehMG 9 months ago
@TehMG I work at a company that manufactures UPS's, rectifiers and power conditioners. I know about shorted semiconductors-They go POW or BOOM! Especially 380 volts and higher they blow violently. Our stuff has fuse links and fuses/breakers but those can't react fast enough to protect an IGBT or SCR that gets switched on at the wrong time or against another device in the circuit failing on it's own. Tap switchers and inverters especially since the devices are in parallel.
Nivicoman 6 months ago
@TehMG With semiconductors it only takes microseconds for a junction to melt and short the layers together. That was an advantage of tubes, a momentary overload or overvoltage didn't make them fail and if they did fail it was usually a burned out heater or loss of emission. Shorted failures were rare with tubes and about the only way they could short was from mechanical shock or extreme overloading and that took time. Cathodes, grids and plates never touch unlike PN junctions.
Nivicoman 6 months ago
Wow somebody shat his pants there :-)
zaprodk 1 year ago
sounds like you need to review the trip settings on your main circuit breakers.
jimbomassoud 1 year ago
@jimbomassoud
I learned that the failure in the UPS was a phase-to-phase short on the input 480 VAC circuit, apparently through the diodes or SCRs in the input AC side. Once the plasma ball happened, the current increased so quickly it amounted to a "bolted fault". I'm told by the electrical engineer of record that breakers don't always coordinate on a bolted fault. I'm happy the one that tripped kept it in that branch circuit rather than going up to the 22kV utility source.
TheDennisChannel 1 year ago
@jimbomassoud The sequence alternator in the buffer setting module may need to be reset to compensating mode. Otherwise, your buffer joint and vertical transformer may overload the interface module in the primary buffer setting of the joint processor in the main shaft ring of your third sequencer.
Make sure you update your third shaft ring sequencer as well by uploading the current firmware.
backinthecrystal 10 months ago
@backinthecrystal That was what I thought. Thanks for supporting me on that!
TheDennisChannel 10 months ago
@backinthecrystal I believe that a Allen-Bradley retro entabulator should have been considered for installation between the vertical transformer & the buffer joint. For more information, you can view the Allen-Bradley retro entabulator through a YouTube search.
Hope this helps.
gblueslover2 6 months ago
@gblueslover2 But a retro entabulator will not get rid of signal degradation between the charlie joints of the post-linear compensator in the internal buffeting jacket. The buffer joint must be connected to a ventricular pump module that is ramscooped into a redundant kiloquad buffer so that the alignment is no less than three millimeters between the signal processor joint and key buffer jacket ring.
backinthecrystal 6 months ago
mga engot
j12072e 1 year ago
Its all good till you let the smoke out. Remember, you must keep smoke (and fire) in at all times. You're lucky you were looking at the battery monitoring system at that point and not with your face in the UPS looking for the serial number to give to tech support.
SouthDFWrep 1 year ago 2
I hope it was covered by a warranty !!!
narcoti 1 year ago
@narcoti
This device was covered on a maintenance agreement with the manufacturers service organization. It was repaired and has worked reliably ever since.
TheDennisChannel 1 year ago