Leave a Reply

Your email address will not be published. Required fields are marked *

Subscribe to Comments:

26 Comments

  1. Thanks for the post, Rod. I had not seen a good explanation of this issue before. BTW, “hiddencamper” is a frequent poster on another forum I read regularly; whoever he/she is, the posts are always clear & well thought out. Someone who obviously takes their time crafting their posts.

  2. Now that the vulnerability is known, knowledge of it will (or, more likely, already has) spread throughout the industry. If this scenario is not already in the simulators, it will be soon. The result will be improved safety though better plant operator skills.

    I am sure my former employer (Schweitzer Engineering Labs) would be like to contribute the company’s expertise in finding a reliable electrical relay control method to detect this condition.

    1. I am a Mechanical Engineer by trade, but I have worked on a project that was installing some Schweitzer relays and also had a bit of involvement in Open Phase projects for 2 different 2-unit sites, but I don’t recall off-hand if the relays that were being used for Open Phase were from Schweitzer.

      It will only take a minute, so I guess I will go take a quick look in that project file.

      Looks like ABB relays were procured for the first part of the Open Phase solution, rather than Schweitzer, donb. Perhaps some other plants went with Schweitzer relays. I saw a prominent ABB advertisement in the Atlanta airport last week above one of the escalators.

      For the 2 plants that I am most familiar with, there is an initial installation of relays that will be used to record data and a later design change will enable a trip function from those relays……which will be based on setpoints that are determined based on the data that is collected during the first portion of the “fix”.

      That is my best recollection of the project as a Mechanical Engineer, and from Rod’s write-up, I am pretty sure that the data collection is being done to prevent the “false positives”.

    2. Compared to many issues – This looks like a pretty easy fix. This looks like an issue that EP would have some input. The SEL blue relays could certainly handle this issue with some logic in their trip equations.

  3. donb: Just so you know, Schweitzer Engineering Labs has been engaged with the efforts at Exelon Generation since the issue was discovered. One of their many relay designs is key to the protection system. While detecting and separating from the Grid on an open phase is important, not separating the Offsite Source from the Plant on a simple Grid imbalance would also not be good. The Schweitzer Relay allows algorithms to be developed and tested that do both.

    1. Steven, you said:
      “While detecting and separating from the Grid on an open phase is important, not separating the Offsite Source from the Plant on a simple Grid imbalance would also not be good.”
      Is this “grid separation” actually what you meant to say? Or rather isolate the fault from the grid (or the station independent power trains), which can be done without “separating from the grid.”
      The difference is no current US NPP can sustain a total grid separation anymore without a reactor trip because it can’t survive the load rejection. I suspect the goal in a solution here is to separate the fault from the system, not separate the plant from the grid since the grid system has multiple supplies and the in-house system has two independent trains. The goal in all electrical fault protection is isolate the fault from the rest of the system.

      1. The whole open phase thing is about getting the vital busses separated from the fault. How this is done depends on the plant.

        In general, this would trip the main or reserve feed to the vital bus and force a transfer to either the alternate source or to the diesel generator. The goal isn’t to trip the main generator, however there may be some fringe cases where that could happen depending on bus layout.

        Side note: I understand a lot of plants can’t survive total load rejects, but I’m not really sure how/why that capability was lost. I know BWRs in the US never licensed GE’s load reject solution (it also would have adversely impacted fuel thermal limits). I’m guessing a combination of thermal limits, fuel duty, EPU, led to most PWRs losing this capability as well.

        I know TMI and some of the B&W plants can withstand a certain level of loss of load for a short time, if the generator output breakers don’t trip open. But that’s different from a full on load reject.

  4. Is there something missing from this story? Does the NRC think that there can be ZERO events at a NPP? Are NPPs to be designed and/or upgraded to operate their entire life with ZERO events? How is this event going to prevent the “Vital Bus” (of which there are two and you only need one to protect the plant) from operating?
    Have these NRC “engineers” been trained on Design Bases Events (DBE) requirements? How may times does the similar event occur at a conventional power plant (coal, NG, Hydro)? Is there any possibility that because of this “Open-Phase” event that the NPP will exceed the design bases licensing requirements of the plant as described in the Technical Specifications? Is this even a Beyond Design Bases Event?” [ If it is a BDBE it is a non problem.] Next, what will actually happen to the plant if/when this actually occurs? What other “BDBE” will be introduced because of the addition of these new “event” triggers – all of the extra relays, sensors, needed maintenance, etc. Finally, since this is outside of the NSSS why is the NRC involved?
    Are they going to erect screens to prevent birds from flying into the HV lines? Are all HV lines going to have de-icers to prevent ice buildup to prevent loss? Are all HV lines going to be placed underground in tornado/hurricane areas to prevent line loss? Then what about floods if they are buried? And I can think of a thousand other “events” with equal or greater probability of causing a loss of power to “vital” loads. Doe each of these need to be addressed?
    40 years ago, due to my Navy Nuclear power experience, I was amazed when I learned that if there was a problem at a civilian NPP, the operators shut it down and fixed it. That was it. When fixed they started it back up, Sometimes within the same day or even the same shift.
    Today, an operator inadvertently flips the wrong switch, the plant trips and a week later the plant owners are still explaining to the NRC how they will prevent (not minimize) future occurrences. Classes are held and simulator training is given on double-double checking and other absurd techniques to prevent “Inadvertently” flipping the wrong switch or pushing the wrong button.
    And you wonder why nuclear power is so expensive in the USA.

    1. @Rich

      Is there something missing from this story?

      Absolutely. I have not tried to address many of the kinds of questions that you posed in one place, but I have provided some links that provide deeper explanations.

      1. Yes, this problem has been going on for a few years. ALL NPPs were to addressin in an NRC Information notice (March 2012) and an NRC Bulletin (July 2012). Plants were required to confirm that licensees comply with design basis (read NRC Bulletin 2012-01 for the full list of items to be addressed)
        As I implied above someone/group wants an awful lot more than to “Comply with design basis.” It is now even on HuffPo, wonder how/why it got there?

        Better info can be found on these links for those interested.

        http://www.nrc.gov/reading-rm/doc-collections/gen-comm/info-notices/2012/ml120480170.pdf

        http://pbadupws.nrc.gov/docs/ML1207/ML12074A115.pdf

        http://pbadupws.nrc.gov/docs/ML1316/ML13169A330.pdf

        http://pbadupws.nrc.gov/docs/ML1432/ML14328A621.pdf

        http://www.huffingtonpost.com/roger-witherspoon/dangerous-flaw-threatens-_b_9382364.html

  5. The FAA takes years to release service bulletins to airlines to fix cracks in structure that would affect safety if they get too big. The bulletins often do not require action after the airline receives them for several more years. The public, if they knew, would be appalled because of their ignorance. The engineers have evaluated the crack growth rates and their impacts. The difference is that there are no anti-airliner groups calling for the end of air travel because they would be laughed off the face of the planet. Nobody is giving up their trip to Hawaii. And like nuclear energy, air travel is the safest at what it does, even though it kills far more people than nuclear.

  6. Transmission / Distribution is fairly new turf to the NRC.
    It became a bigger deal with deregulation, when distribution grids became overtaxed by over zealous power traders.
    I noticed back in the late 1990s. ComEd and Union Electric were trading large amounts of power between St Louis and Chicago, across the lines my power plant (Clinton) was connected to. Generating 985 Mwe (at the time), the reactive load would hit 400 MVAR on a hot summer afternoon. If you’ve never been in a Control Room under extreme reactive load conditions, it looks like this. The generator voltage regulator high limit warning lights are on. Alterex High Temperature alarm comes and goes. Generator stator hot spots require constant attention, and on top of that balance of plant stuff – Safety Related 4160 VAC Bus Voltage lowers towards 3900 VAC, and degraded voltage timer relay setpoints are not far below there. We kept one off site power transformer on the high taps, so that we could swap Safety Buses to a higher voltage source under such conditions. In the evening, we’d swap buses back to the lower tap transformer. Lots of breakers being swapped.
    One evening, while doing just that – our NRC Senior Resident wanted the ten minute version of what we were doing and why. We have Tech Spec LCO 3.8.1.1 that requires operable Safety Buses, and if the grid variation pushes us, we’ll swap sources to keep our Bus Voltages in range. NRC doesn’t control the grid. There’s not one 345 kV or 500 kV voltage span mentioned in Tech Specs. But there is for 4160 VAC and UV relay setpoints.
    That plant has since been fitted with Static VAR Compensators, and doesn’t get the pillar to post voltage swings of yesteryear.

    With Open Phase – the issue happens outside the plant, on the grid infrastructure. At Byron, the indications triggered operators to take appropriate action. The NRC doesn’t like to rely on manual actions when they can force new relay sensing and logic into the scenario.
    Operators are taught that when a phase opens up in a three phase system, loads continue to operate with reduced capacity. The third phase is a conductor moving in a magnetic field and it will therefore indicate some voltage, well below normal.

    The grid has ISOs, FERC, and NERC, and its about to include the NRC in its regulatory arena.

    1. The next problem will be “Is this added equipment Safety Related or even Important to Safety, or just the typical balance of plant equipment.” The problems introduced by making this Safety Related not only increases the cost of the equipment by an order of magnitude, but creates even more problems. Then the NRC will want the previously non-safety related buses made Safety Related. This will mean Safety Related service, towers, capacitors, disconnect switches, insulators, transformers, etc., etc. Then what of all of the Non-safety related equipment fed by this now Safety related mini-substation? Supposedly Non-safety related equipment cannot be fed by a Safety Related system unless there are dedicated breakers to isolate the non-safety related system. This will create another problem similar to the original. Does this mean a separate “Non-safety” aux transformer(s)? Then the NRC will realize that there is not a true Safety related system like the Reactor Protection System, Emergency Core Cooling System, etc. that has 3 or 4 sensors on each parameter monitored with 3 or 4 separate safety related trains and a safety related logic system “Protecting” us from this event. More clicks of the ratchet.

      If the NRC can force this equipment to be installed then, it is Safety related and the ratchet wrench starts ratcheting. And the NRC ratchet wrench gets tightened several more clicks. The mountain they are building far exceeds the potential damage of the mole hill they discovered.

      And, speaking of. Google “United Airlines Flight 232.” This would have been a “non-event” if the hydraulic systems had a “Excess Flow Check Valve” to prevent the ruptured hydraulic system from pumping out all of the oil from the two operating hydraulic systems. 111 dead, 47 seriously injured and here it is 25 years later and there is no FAA requirement for this system. Yet NRC “regulations and/or guidance” requires this or other means to prevent loss of a hydraulic system in this fashion when a common reservoir is used.

      1. The NRC and industry were back and forth on the open phase trip system. The system is non-safety, because it is a power grid/infrastructure system. However it will be in tech specs per criterion 4 I believe, due to the substantial risk an open phase condition can cause and the fact that it is necessary for General Design Criterion 17 compliance.

        The trip system will be required to have some quality level assigned to it, but not full on safety-related.

        The reason this is important, is because both/all ESF busses could be getting power from the same source when the open phase condition occurs, causing a potential common mode failure of the ESF busses like what Byron experienced for 8 minutes.

    2. Clinton’s SVCs were actually a godsend for the open phase event. The SVC can accurately detect open phase conditions and will trip on one. Meaning an SVC trip is a potential indicator of an open phase condition. So for Clinton, the comp action is if the SVC trips, immediately verify vital bus voltage across all three phases and force a transfer to the alternate feed source if there is an open phase. These actions were already being performed by operators anytime an SVC tripped, so for us it was just about writing it down in a procedure.

      Other plants without an automatic alarm or detection scheme had to do a hell of a lot more to maintain OPERABILITY of the offsite power system.

      I’m an SRO at Clinton.

      1. @Michael Antonelli

        Thank you for the information. Can you translate “SVC” for me.

        For all other readers, SRO is Senior Reactor Operator (not Shutdown Reactor Operation as on a submarine.)

        1. Sorry! The industry is an acronym nightmare!

          SVC = Static Var Compensator.

          Rob Brixley was talking about voltage on the vital busses in the 90s drifting all over the place during summer time. The conclusion to that story, is a study was eventually performed which determined after a plant trip during high MVAR loading conditions, the loss of the Clinton main generator could render the 345kv offsite power source to the vital busses below the minimum voltage for OPERABILITY, and in some cases, even cause the 138kV source to drop. The NRC claimed this was a violation of General Design Criterion 17 (GDC 17), the same critera that the open-phase issue is being worked under.

          To deal with this issue, Clinton installed two Static Var Compensators, one on the plant side of the 345kV system transformer that feeds the vital busses, and the other on the plant side of the 138kV system transformer. The SVCs use capacitors and reactors to provide roughly a 30-40 MVAR control range for the vital busses which will automatically switch in and out to ensure the vital busses have sufficient voltage from the offsite power system.

          As part of licensing the SVCs which are non-safety, the SVCs included pretty complex electrical protection, which ensure if the SVC starts to misbehave, it will trip off to at least protect the vital busses and prevent a common mode failure. The SVC electrical protection system is actually part of the plant’s Tech Specs. So fortunately for us, the SVC protection system is capable of detecting single phase conditions and will trip the SVC off, which means the SVC trip alarm in the control room will directly alert us that we may have an open phase condition. As I said before, we take the same actions that we would before the open phase issue existed, verify vital bus voltage and transfer/adjust as necessary to maintain OPERABILITY, the only difference is we “train” on it and have a “designated comp action”.

          Some plants had to go to much greater lengths to demonstrate to the NRC that they could detect an open phase condition. We see it right when it happens with existing systems.

  7. hiddencamper said it is dangerous to an operator to trip the breaker locally (for a condenser cooling water pump). This is new to me that it is dangerous to trip locally -at the breaker? why is this so?

    1. The electrical safety requirements have changed considerably. Opening and closing breakers locally is expected to be a last resort, especially for vital busses. Most plants including my own only rack 4160V or above breakers using robots. For close/trip, if I have a breaker fail to trip, it’s preferred to dump the entire bus than open the breaker door and manually hit the trip latch.

      The issue is a fault can cause a substantial arc. The vital busses have relay settings such that a fault can be sustained for a long time (in electrical timeframes) to allow the bus to survive an ECCS block start without relaying tripping the bus. This means if an actual fault occurs, a substantial amount of bus work will arc prior to the feed breakers opening.

      The minimum approach distance to prevent 3rd degree burns for my plant’s high pressure core spray bus is 140 feet. Literally the entire room is dangerous if the bus faults during opening and closing. So we evacuate the room prior to performing breaker switching. The other vital busses are like 63 and 72 feet. The non vitals are more like 20-30, and if you were in full class 3 electrical safety gear you wouldn’t get more than 2nd degree burns (assuming the concussive shock didn’t knock you out first).

      99.99% of the time a breaker will close/trip properly. But the nuclear industry has taken the approach that it’s not worth locally operating a breaker outside of emergencies because of this potential risk.

      1. @Hiddencamper

        Thank you for the explanation.

        Speaking as a former submarine engineer officer who was an observer during manual breaker operation, I can’t help but be impressed and saddened at the same time.

        It sounds like a number of real physical risks have been inserted into your plant’s design in order to prevent a hypothetical risk of a double ended break of a large diameter primary coolant pipe with subsequent rapid blowdown and need for immediate start of emergency core cooling pumps.

  8. When 4160 or 6900 VAC breakers are opened, an electrical arc in all three phases is suppressed in the breaker by a component known as an arc chute.
    Some breakers use a mechanical bellows driven puff of air to blow the arc into the arc chute, most use magnetic windings to draw the arc into the arc chute for dissipation.
    If a fault exists a much more energetic arc must be broken and dissipated. I have locally tripped a 4160 VAC Condensate Pump in my younger days, a control room switch failed to stop the pump. When I depressed the TRIP plunger, the KACHUNK sound wasn’t nearly as unnerving as the flash of blue-white light that was under the breaker. Distance is your friend.

      1. Thanks Rob and Rod. I have had plant experience in startup and operations, but had not been in a plant where this happened,

        Rob, did this hurt you or just scare you? I hope not hurt. If just scared it seems it is something that should be in Operator Training.

        Howard

        1. I have an abiding respect for electricity, going way back to my Navy years when I was an Electronics Technician.
          Zaps, the flash of an arc, and the smell of Ozone reinforce that respect. Operators are trained on local operation of breakers, safety precautions are emphasized.

          I always hated operating 345 kV High Voltage disconnects out in the switchyard the most. Seems like ice storms and severe weather always go along with having to isolate a power line.These have a hand crank for operation. When the buzzing and crackling arc forms and dissipates, you can see your shadow in the gravel skitter around as a result of the arc, the light source, dancing around – even at high noon. You remember “don’t stop cranking” until that arc breaks. They make Motor Operated Disconnects (MODs) you don’t have to crank, but they are mostly reserved for big ticket items like Main Generator Output disconnects.

          1. Agree, HV switch yard operations could be nerve wracking. Most everything you touch would give you a “poke”; explained to us as static electricity. We were told “don’t be timid, grab on forcefully.” The only hand crank devices we had were the grounding devices, which were cranked closed on isolated sections. Ice storms were especially bad as when they start to thaw 10 foot ice sickles could form on the HV lines all over the yard. Randomly dropping as they thawed. Also if the ice formed a path to ground they would “blow free.” Our Air Blast Breakers (ABBs) had a mechanical cycle counter inside the air compressor enclosure. On real severe weather events, with a lot of fault clearing and “auto reclosures” on some faults, the Load Dispatchers always wanted the cycle counts. On some faults (determined by the relay “targets” we reported) he was allowed to try closure again, up to a limit of tries. But you always knew you were going to get a “poke” during these operations. I guess it is the nature of HV at 345KV. It can induce a static charge on anything near it, and will use anything it can for a ground, including us.

Recent Comments from our Readers

  1. Avatar
  2. Avatar
  3. Avatar
  4. Avatar
  5. Avatar

Similar Posts