A Blocked Toilet Led to $500M in Lost Revenue
Summary
A plumber called to repair a blocked toilet triggers a series of events that lead to a catastrophic failure of the automation system and an excess of $500M in losses. Little did he know what a mess he was inheriting. But, the main problem was caused by inadequate protection of the PLC’s and ignoring the NEMA procedures for water-damaged control equipment.
Read this and wise up. It's pretty amusing (for us, not them).
Failed Prevention Methods Lead to the Blame Game
Poor management, poor maintenance, and a breakdown in communications can be disastrous for any business. This proved to be the case when a power and gas-fired power plant exploded in the middle of the night in the midwestern United States. Various circumstances, unfortunate events and human error resulted in one of the most powerful explosions and a massive loss of revenue. By examining the steps that led to this imminent disaster, it provides insight into the value of taking precautions and having open communication between management and operations to avoid preventable mistakes.
The explosion led to a laborious litigation in which the findings presumed the explosion was primarily caused by the utility. In fact, everyone had a part to play and this included management for failing to recognize the potential for catastrophe, not training technicians adequately, avoiding engineering support, as well as operations for not responding to obvious warning signs and not enforcing procedural repairs in a timely fashion. Finally, maintenance was to blame for not following protocol during any repairs that were actually done.
Some blamed the installation of a PVC wastewater line that was installed a few years before the explosion happened. It was installed too close to the blowdown tank outlet pipe and eventually collapsed from the heat given off of the outlet pipe. This resulted in sewage backups and floods. Even though it was replaced in one section, this failed to solve the greater root cause of the pipe collapse.
Overlooking Problems Create a Sewage Disaster
The problem started decades ago: Initially, an undersized steam boiler was installed many decades previous. This was overseen by management at the time and if they had done the due diligence, they would have found it to be too small to release the steam that needed to be released in the event of an emergency shutdown. As a result, the outlet pipe from the steam blowdown tank carried steam instead of condensed water, resulting in damage to the concrete outlet pipe.
Fast forward to 2003. Now picture this: Mr John Doe Plumber (or sewage contractor if you prefer) gets called in to the plant to attend to a backed-up toilet in the power plant control room. He goes about his business (as plumbers do) attempting to clear the clog and his jetting tool becoming lodged in the check valve. Little does he know, that a few days before, the wastewater line had collapsed again. Part of the wastewater system included a pumping station that pumped through the same damaged wastewater line. To implement this, a check valve was inserted in the leg of the line extending to the control room toilet. No plant personnel locked out the lift station pump when the check valve was blocked open. So, a short time later the automatic pump turned on and, due to the open check valve, discharged an estimated 200 gallons of raw sewage through the toilet pipe onto the third story control room floor. All in a days work for Mr John Doe? Poor guy.
As the sewage flowed onto the third story control room floor, more of the sewage was draining along the cabinets of the PLC that controlled the fuel safety system. The PLC system is vital to the correct operation of the boilers, igniters, blowers, which monitors and controls the system with accuracy. The sewage was slowly flooding the PLC system and also, the plant operators didn't close or lock-out/tag-out the main gas valve for the boiler as their own procedures required! So, with all that raw sewage, the boiler wasn't firing, and the water damaged a number of PLC components, causing the PLC to go into a fault condition.
When maintenance was called to assist with the cleanup and repairs, they made a number of errors that increased the potential for disaster. This included assigning task technicians with no prior experience to work on the FSS (Fuel Safety System). Their solution? Wipe the affected devices in visual range and blow dry them with compressed air. Completely avoiding the recommendations of National Electrical Manufacturers Association (NEMA), “Guidelines for Handling Water Damaged Electrical Equipment,” which would have required them to replace the components and/or return them to the manufacturer for condition assessment.
Faulty PLC Components Led to an Explosion
The next error was to incorrectly reset the hardwire trip circuit Although this didn't cause the gas valve to open at that time, it did terminate the “close” signal, which would have overridden any inadvertent “open” signal. Components of the PLC for the fuel safety system continued to be inundated with the raw sewage. Controller cards were swapped resulting in misinterpreted incoming signals between Rack 1 and Rack 2. This set the stage for a signal intended to ensure that a valve remained closed being redirected to open the main gas valve about three hours before the explosion.
As gas was released into the boiler, the mixture of air and gas finally reached a critical point at the igniter. About two and a quarter hours later it ignited an explosion that completely destroyed the boiler and damaged other plant systems. The explosion destroyed the boiler of the 450MW coal and gas-fired power plant, resulting in physical damages and lost revenues estimated to be in excess of $500 million. Luckily, not a single person was killed.
This particular explosion could have been avoided if one person in the chain of command had done their job. The explosion was entirely preventable. At any point in the process, management, operations, or maintenance could have interrupted the chain of events that ended in disaster with care, thought, and adherence to well-designed procedures. Suffice to say, it remains clear why open communication, chain of command and accountability are vital aspects of an efficient and well-oiled machine (power plant in this case!). Also, if you are a sewage contractor, insist on historical drawings of complex systems when dealing with sewage!
Note: This rewrite is based on an article written by John A. Palmer, Ph.D., P.E., C.F.E.I. and David A. Danaher, Knott Laboratory, Inc. | Nov 01, 2004 The original may be found here.