Mitsubishi Heavy Industries (MHI), the supplier that sold four new steam generators to Southern California Edison (SCE) for the San Onofre Nuclear Generating Station (SONGS), has issued a redacted version of its root cause analysis of the u-tube failures that have kept both of the station’s 1100 MWe units shut down since January 31, 2012. My analysis of the report tells me that a large, skilled, experienced team of engineers (both at the supplier and at the utility) made a design choice that resulted in unexpected and unintended consequences.
Early last month, two technically unqualified politicians (that phase is a bit redundant, isn’t it) – Senator Boxer and Congressman Markey – took it upon themselves to demonize selected nuclear energy professionals. They extracted a few isolated phrases from a version of the root cause analysis that was not publicly available and proclaimed to the world that they had found a smoking gun “proving” that SCE had knowingly installed faulty equipment.
Aside from the fact that such an assertion was absurd – why on earth would any corporation take the risk of installing components known to be faulty into a vital, multi-billion dollar production facility capable of producing between $1-$10 million in daily revenue – it exposed a visceral dislike of a power source that has been cleanly and safely supplying 20% of the electricity in the United States for several decades.
It also exposed a profound distrust of one of the most squeaky clean industries in the United States; say what you want to about nuclear energy, but it does not take much time in the industry to realize just how differently it is led compared to all other money making enterprises.
One of the major difficulties in this saga is the fact that politicians rarely understand engineering, especially the constant need to make informed decisions and to balance competing requirements. No mechanical system is flawless and no material is perfectly matched to its environment. That statement is especially true when the environment is a complex heat exchanger required to operate over a wide range of temperatures in a variable mix of fluid conditions over several decades.
After many months of investigation, tens of thousands of hours of analysis, and hundreds of millions of dollars worth of lost production time, it is now clear that the two steam generators installed in San Onofre Unit 3 contained a minor manufacturing feature that resulted in a “perfect pitch” harmonic. At just the wrong condition – 100% steam flow – a combination of relatively dry steam, precisely manufactured anti-vibration bars (AVB), and densely packed u-tubes resulted in a few hundred (out of nearly 10,000) tubes vibrating with a large enough amplitude to make contact. The unexpected vibration and contact resulted in accelerated wear and caused one tube to fail while the steam generator was operating.
If Unit 3 had remained in operation, other tubes in the same area of the steam generator would have likely failed. The same problem does not exist in Unit 2.
(See MHI’s document number L5-04GA588(0) titled San Onofre Nuclear Generating Station, Units 2 & 3 Replacement Steam Generators Supplementary Technical Evaluation Report Fig 2.1-1 (1/2 and 2/2) and 2.1-2 (1/2 and 2/2) to understand the basis for that statement.)
Update (Posted on March 10, 2013 at 08:25) The full document Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generators of San Onofre Nuclear Generating Station is now available from the NRC ADAMS database. End Update.
Surprisingly enough, the reason that the condition does not exist in Unit 2 is that the anti-vibration bars (AVB) in Unit 2 were made with enough less precision that they prevented the perfect pitch situation. Instead of being virtually perfectly round holes through which the steam generator tubes could penetrate with tight tolerance but no contact, the AVB’s in unit 2 had enough manufacturing variation that they made contact with the penetrating tubes with an average force that was twice as high as the minor, incidental contact achieved in Unit 3.
All of the TTW tubes are located in the region of highest average void fraction, where velocities are highest and damping is lowest. Both Unit 2 and Unit 3 have the same thermal hydraulic conditions. The tube-to-AVB contact forces in the Unit 3 RSGs are smaller by a factor of two than those of the Unit 2 RSGs. Almost all of the TTW tubes were found in the Unit 3 RSGs. The difference in the contact forces explains this large difference between the two units.
(Section 2.4, Conclusion)
That extra contact force, which was considered to be undesirable by the designers at the time they designed and manufactured the tubes, provided enough unplanned disruption to the tube bundle that the harmonic vibration could not get started and could not reach enough of an amplitude to cause tube to tube wear (TTW).
It is instructive to learn that the tighter tolerances in unit 3 were purposely chosen because the supplier was seeking continuous process improvement. MHI engineers had determined that a small change in the manufacturing process could improve the repeatability of the AVB holes. The design team agreed that the tighter tolerances resulted in a design that was “significantly more conservative than previous designs in addressing U-bend tube vibration and wear.” (page 48 of MHI’s root cause analysis)
Because the computer models used for the design process were not perfect fidelity reproductions of the complete environment of the steam generator, simulation runs did not reveal the potentially detrimental effect of the tighter tolerances.
Though it does not come out and say this in the report, I believe it is likely that the cause of the slight simulation infidelity condition was a limited data set. The San Onofre steam generators are different from any other steam generator that the company has produced; the only ones that are remotely similar are those at Fort Calhoun. Even those generators have major differences, including substantially different sizes. MHI has manufactured dozens of nuclear power plant steam generators that are reliably operating around the world, but they use a design that could not meet the requirements for installation into the San Onofre units.
Misunderstood technical details
The phrases that Boxer and Markey used to assert that there was malfeasance on the part of the utility and the supplier came in places where the root cause analysis team was discussing the design process and the reasons why certain options were determined to be less than optimal. As any reasonably experienced engineer knows, design is fundamentally about tradeoffs among many different measures of effectiveness. A choice that might improve performance in one area often imposes a major degradation in other areas – what the leaked phrases referred to as “unacceptable consequences.”
It is simply part of the design process to propose, evaluate and reject several different alternatives, even if they appear to have advantages on one or two measures. In fact, design is fundamentally the process of rejecting an entire universe of alternatives and picking the ONE that is determined to be the best choice when all measures are considered.
I am sure that, with the advantage of hindsight, the engineers at MHI will never again produce a steam generator that experiences tube to tube wear (TTW) caused by nearly perfect anti-vibration bars (AVB) in a steam generator with high quality steam. I am also sure that human engineers will continue to produce better and better products that occasionally fail to perform as expected. Just look at the Boeing Dreamliner situation or even your own last automobile purchase for examples of fine engineering that is not quite as perfect as we might like.
There are many aspects of the San Onofre saga that sadden me. It is a terrible waste of resources; operating fossil fuel powered replacement generating plants has made the air dirtier and the people in Southern California a little less safe; the situation has given people like Markey and Boxer another reason to battle against the use of nuclear energy (despite the fact that there was no public health risk or impact); and decisions made in the wake of the leak have put an important area of my home country at risk of severe power disruptions in the near future.
Most of this could have been avoided if there was not such a culture of nuclear exceptionalism in which, of all energy sources, only nuclear must be perfect. It should be enough that nuclear fission is far superior to all other choices, but we still have a long way to go before the public will accept that message. We are not being well led by technically ignorant politicians and we are not being well served by a culture of “zero tolerance” within the industry itself.
Both of the units at San Onofre, by all rights, should be operating today, supplying reliable, emission free electricity. Instead, there is no current hope that the start up will happen any time soon. The replacement fuel pump meter continues to run and fill the coffers of the competitive fuel suppliers. The 700 or so SONGS employees who have already been laid off will most likely be joined by more colleagues in the near future.
It is too idealistic to hope that either Markey or Boxer will decide that they should apologize for their accusations and their continued efforts to muddy the waters and delay (or prevent) the restoration of two of the most useful and safe electricity producing units in Southern California.
Rest of the story
I think it is important to remind everyone exactly what happened. On January 31, 2012 (more than 13 months ago), San Onofre Unit 3 operators received indications that one of the two steam generators of the plant they were running was leaking and causing a tiny, but measurable, increase in the radioactivity of the normally non-radioactive water in the secondary (steam) side of the steam generators.
When they recognized the indication, the operators took the conservative course of action and shut down the nuclear plant, even though it turns out that the leak rate (84 gallons of coolant per day) was below the plant’s allowed technical specification (150 gallons of coolant per day per steam generator)
Aside: For the true nuclear professional geeks reading this, the reference for that number is LCO 3.4.13, RCS Operational Leakage. End Aside.
According to information obtained several months later, the MAXIMUM potential dose of radiation to anyone was 5.2E-5 millirem (0.000000052 rem) which is one billion times lower than the annual limit for radiation workers at the time that I first became a nuclear energy professional.
I would also like to remind people that steam generators have periodically experienced leaks since the very first pressurized water reactor was built. The devices do a great job of keeping the slightly radioactive primary coolant contained. They are important on board submarines, where there is no place for any radioactivity to go, but on land, steam generators are more of a choice than a requirement. About 1/3 of all light water reactors do not even bother with them, boiling water reactors send primary coolant directly to the steam turbines.
The small u-tube leak never represented any risk to the public. Unit 2 was not operating at the time and never experienced any steam generator u-tube leaks. Neither unit was in violation of its operating license; before submitting a recovery plan to the NRC that committed the utility to a course of action that included asking the NRC for permission to restart, there was no legal requirement for the plant operators to ask permission to fix their plant and start it back up again.
Unfortunately, nuclear plant operators, perhaps especially those in areas represented by politicians that will take every opportunity they can get to criticize the technology, run their companies with an abundance of conservatism. They believe that they will be better off practically genuflecting in front of regulators than by aggressively defending their rights to properly decide how to run their power plants.
I know I do not speak for my employer when I say this, but that is often not the correct or safe course of action. The regulators have a job to do; they need to ask hard questions. However, the plant operator should know far more than the regulator and should willingly accept all of the responsibilities associated with owning and operating electrical power plants in the safest and most reliable manner possible.
As this situation and several other show, there is a great financial reward associated with safety and reliability.
Atomic Power Review – (March 9, 2012) San Onofre: MHI document release by NRC and what it really means
Yes Vermont Yankee – San Onofre Thoughts and Future. I told you so.
Idaho Samizdat – Reactions to reactor restart remarks about San Onofre