In the November 1998 issue, PEi reported some key findings of a NERC report on the North American electrical system readiness for Y2K. In our third in this series of articles, we examine the importance of communications to the industry and NERC`s latest step in its rigorous Y2K programme – a drill simulating the partial loss of utilities` communication capabilities.
As the countdown shows fewer and fewer days left until the critical rollover from 1999 to 2000, entire industries and individual companies are becoming increasingly aware of what the new Millennium could mean for them. For some it could mean loss of normal operations resulting in anything from mild confusion to complete chaos, while for others it could mean business as usual. Either way, it is likely that some form of preparation or testing has been done, however basic or extensive.
One industry falling under the “extensive” category is the North American electric power system, from generator to supplier, which has for the past 12 months been reporting to the North American Electric Reliability Council (NERC) on Y2K readiness. In addition to planning, assessment and testing of their systems, these utilities and plant owners have been examining back-up systems and making contingency plans. An important part of this was a recent “Year 2000 drill” simulating the partial loss of voice and data communications needed to operate their systems.
The drill, carried out on April 9, involved around 200 utilities in the USA and Canada and over 2000 personnel. For several hours the utilities simulated a Y2K communications failure by using back-up voice systems and manual procedures needed to operate the electric system. The drill allowed NERC to identify areas where systems and procedures require further attention.
NERC, which coordinated the drill and is coordinating the industry`s Y2K programme, has continually stressed the importance of the telecommunications sector to the electricity industry. It will implement the lessons learned from this drill in a second drill to be carried out in September.
Under the auspices of the Department of Energy (DOE), NERC is responsible for coordinating and implementing the Y2K readiness programme for the electricity systems of the USA and Canada. It reports quarterly to the DOE on progress made in its Coordination Plan, and submitted its third quarterly report at the end of April 1999.
The goal of NERC`s Y2K coordination plan is to prepare the North American electric systems for reliable and sustained operations into the Year 2000 and beyond. It aims to do this through several key objectives:
à¢€¢ Assuring that mission-critical systems are Y2K ready by June 30, 1999 through co-ordination of a rigorous programme of identification, testing and remediation of vulnerable systems.
à¢€¢ Coordinating the sharing of Y2K technical and project management information and resources through industry conferences, technical committee meetings and the NERC website.
à¢€¢ Coordinating the assessment of Y2K operational risks and developing and implementing contingency plans.
à¢€¢ Coordinating industry readiness drills.
NERC measures Y2K progress as a percentage of work completed in three key phases: Inventory; Assessment; and Remediation and Testing. These terms represent a reasonable division of Y2K technical work, but their exact definition is flexible in the NERC programme to allow utilities to fit the work in with their own internal programmes.
About 98.6 per cent of the 3088 electricity supply and delivery organizations in North America have taken part in NERC`s Y2K readiness process to date. All bulk electric entities and control areas, and about 98.6 per cent of the 2888 distribution entities in North America have participated, and NERC has set a completion deadline of 30 June 1999.
But in spite of such a positive and comprehensive response from the industry, NERC is well aware that the Y2K bug is a very real problem. NERC`s awareness of this has led it to create a “defence in depth” strategy, which assumes that although all reasonable and necessary preventive steps have been taken, it can never be 100 per cent sure that major system failures will not cause a catastrophic outcome. Defence in depth thus seeks to establish multiple barriers to reduce the risk of catastrophic events and to mitigate their severity.
Contingency planning is therefore an important part in NERC`s plans, and of key importance is the telecoms sector. Given any failure in the electric system, utilities, plant owners and control centres within and between regions must be able to effectively communicate with each other using back-up systems.
Although the electric industry owns and operates much of its communications equipment, there remains a portion which is dependent on local telephone carriers, long distance carriers, satellites, cellular system, paging systems, network service providers and others. The industry is therefore dependent on a complex set of integrated communication systems.
In testing their communications systems and reporting to NERC, most utilities have needed support from equipment vendors and other suppliers. In the communications area, the inventory phase is now 99 per cent complete and the assessment phase 93 per cent complete. NERC has reported that most systems will be ready by June 30.
Examples of areas where Y2K anomalies have been discovered in electric utility-owned communications systems include:
à¢€¢ Network management software.
à¢€¢ Routers: primary functions work, diagnostics software may be affected.
à¢€¢ Control signalling unit/digital signalling unit devices: incorrect date display.
à¢€¢ Some PBXs may require remediation.
à¢€¢ Incorrect date stamp on fax machines.
Data communication provides real-time updates of electric system status to SCADA systems in distribution and bulk electric control centres. It is also used for remote control of devices in the field, and allows generating units to follow the real-time control signals from the control centre that are needed to instantaneously balance demand and generation.
Although the industry is confident that there will not be widespread interruptions of data communications, it is preparing for this possibility as an unlikely but critical contingency. The main strategy is to operate using manual transfer of a minimum set of critical information over redundant voice systems such as microwave, radios, satellite voice systems, and privately owned telephone networks. NERC recommends that back-up voice communication systems should be used that do not have common failure modes with primary systems.
This critical information typically includes power flows on key transmission lines, voltages and interconnection frequency. Qualified field personnel at critical substations can read this information locally and convey it to system operators in control centres. Essential information relating to balancing, frequency control and transmission security functions can therefore be sustained. A similar type of operation can be performed in electric distribution systems, although the focus is more on switching activities.
The drill scenario
It was this type of back-up activity that was simulated by North American utilities on April 9. A typical drill scenario included the loss of external voice systems and real-time data systems which are used to monitor and control electric power systems. Additional aspects of the drill included partial loss of data acquisition and control computers in the energy control centres.
Several hundred generating units took part in the drill, including coal, gas, oil, hydro and nuclear facilities. Over 500 critical electric facilities in North America were staffed with field personnel using backup radios, privately owned telephone networks, microwave systems and satellite voice systems. These personnel relayed critical information such as power flows, voltage and frequency to control centres to allow critical-system operating tasks including generation dispatch and energy interchange.
Some of this information from substations and power plants was updated every 15 minutes to allow the simulation of continuous monitoring and control. In addition, the drill included inter-area coordination across regional boundaries and included a link to NERC and DOE personnel in Washington, D.C.
The overall aim of the drill was to implement an industry-wide exercise to demonstrate the ability to operate the bulk electric system with limited voice and data communications and reduced EMS and SCADA functionality. The drill was run in parallel with and was kept entirely separate from normal utility operations. NERC`s Y2K Contingency Planning Task Force identified five areas to ensure the successful implementation of the drill.
à¢€¢ Review existing internal procedures to maintain critical data and voice information exchange in the event of the loss of one or more critical telecommunications facilities.
à¢€¢ Identify critical voice and data telecommunications channels and determine the information necessary to continue operation.
à¢€¢ Dispatch personnel to key locations to allow continued monitoring and operation.
à¢€¢ Have personnel identify, obtain and communicate key operating information sufficient to monitor and operate the system according to procedures.
à¢€¢ Use alternative back-up communication systems for data and voice communications.
Under NERC guidelines, each organization taking part in the drill developed a design team to develop the drill plan and scenarios to be simulated. In developing the drill, the design team considered a number of factors including what communication interfaces their organization uses, what type of communication loss or failure is realistic, and what the impact of a communication loss would be.
The types of scenario conditions that were simulated included:
à¢€¢ Partial loss of voice communication systems, either internally or externally owned.
à¢€¢ Partial loss or data/telemetry.
à¢€¢ Partial erroneous or missing data in EMS, SCADA and control systems.
à¢€¢ Partial loss of EMS, SCADA or control system functionality.
à¢€¢ Partial loss of internet provider services.
The drill assumed that each utility experienced one or more of these problems, and lasted between four and six hours for most participants.
In order to report the findings of the drill, NERC requires each participating organization to hold two evaluation and assessment sessions, the first of which took place immediately after the drill. The purpose of this was to record all key events that occurred during the drill and to detail the actions and decisions of personnel taking part through evaluation questionnaires.
NERC reported that the drill was “extremely valuable” for the training of personnel and practising back-up communication procedures. It identified several “lessons learned” during the drill which it aims to incorporate during the planned September drill:
à¢€¢ Some of the written procedures and checklists had telephone numbers that need to be updated.
à¢€¢ Geographic coverage of radios did not reach all essential facilities and will require relocation of some antennas.
à¢€¢ Personnel require additional training on the use of satellite voice systems.
à¢€¢ Some congestion of voice traffic on some of the back-up systems occurred, indicating a need to establish call priorities, co-ordinate schedules for routine reports and proper radio protocols.
à¢€¢ Some back-up voice transmissions were noisy.
à¢€¢ Some back-up voice systems did not work and will require further evaluation prior to the next drill.
à¢€¢ Severe lightning storms in the Midwest interfered with some radio transmissions.
NERC`s next goal beyond producing its final report to DOE at the end of July, is to implement a similar Y2K drill on September 8 and 9, 1999. This drill will be more extensive and will give the North American electric power systems the opportunity to rehearse key portions of their administrative, operating, communications and contingency response plans for the transition to the year 2000. It will span the hours around midnight and will include added interfaces and complexities to the April drill.