|Power plant automation systems contain a staggering amount of data from disparate distributed control systems, safety systems, measurements and software applications|
Configuration Management (CM) is essential for the safe and reliable operation of any power plant. It is an integrated process that reaches across all areas of plant Operations Maintenance & Engineering (OM&E) including corporate support organisations. The CM process ensures the design bases for plant components, systems, hardware and software are well defined and documented, and are used as the reference criteria for evaluating proposed changes involving repairs, revisions, replacements and temporary alterations.
Management of Change (MOC) is a subset of the CM process for managing those proposed change requests from initiation through to evaluation, management approval, implementation and final documentation. MOC ensures that changes are not carried out at the discretion of individual plant personnel.
MOC is a subject that the process industry knows well. It is a core process for managing modifications to elements such as plant infrastructure and equipment. Although MOC can be time consuming, most acknowledge that it improves safety and reliability. Globally, most countries have also recognised the need to manage change in process plants and have established regulations.
But the proper management of automation changes is generally not fully addressed. Most process plants operate with significant hidden vulnerabilities due to poor automation change management that many of their operators are unaware of. Undocumented and unapproved changes to automation systems have been identified as contributing factors to several process industry incidents and accidents around the world. PAS recently conducted a survey of its petroleum, power generation, and chemical industry clients, asking the question: “Have modifications to automation systems ever led to an incident or process disturbance at your site?” Shockingly, nearly 65 per cent of respondents said they had. This figure reveals that proper management and documentation of automation configuration changes, while essential to control system security, plant safety and reliability, is not being practised as it should be.
Automation systems used by power plants contain an astronomical amount of data from disparate distributed control systems, safety systems, measurements and software applications. Because these systems are primary vehicles for continuous improvement of the process, they are continually undergoing change.
In the same survey, customers were asked, “How often do you make one or more modifications to the configuration of your automation systems?” Nearly 38 per cent responded that they made changes at least once per day. Another 35 per cent responded that they made changes at a frequency of between one day and one week.
We also asked, “Do automation system modifications currently require an MOC process at your facility?” Forty-seven per cent responded that they sometimes do and 53 per cent responded that they always do. Taken together, these three questions tell a chilling story about how automation is managed in these facilities. Automation systems are often modified. An MOC process for those changes is only sometimes required. Automation modifications have led to incidents and/or lost production at nearly two thirds of the sites polled.
To understand why the MOC process is only sporadically applied to automation, we must consider its evolution. MOC has long been deemed best practice by many companies, but it only became law in the US in 1992 when Congress passed the 29 CFR 1910.119 Process Safety Management (PSM) regulation.
This regulation, aimed at preventing or minimising the consequences of catastrophic releases of toxic, reactive, flammable or explosive chemicals, mandated that all manufacturers of these chemicals have a compliant MOC process in place by 1997. Other governments around the world quickly saw the regulation’s benefits and enacted similar laws. In general, most PSM regulations are quite rigorous for physical plant and procedural changes but much less stringent for automation systems.
Another reason MOC is sporadically applied to automation systems is the relative ease of modifying automation system configuration. It is possible to modify most automation systems with significant consequences by simply typing in database parameters. Items such as changes to critical alarm limits, or re-ranging an instrument are accomplished in this way.
As traditional MOC takes much more time and energy than these actual changes, it seems excessive and is overlooked. This is particularly true in the power industry, where plant engineer and technician staffing levels have been reduced by 50–70 per cent while management’s expectations for safe and reliable generation remain unchanged.
But ignoring automation systems opens the door for unapproved and undocumented changes to be introduced into automation configurations, with potentially disastrous results. It should be noted that, in contrast, a robust MOC process is almost always followed if an automation configuration modification affects a drawing, procedure or any other entity that is already under change control.
Real world example – FGD trip
An example of an incident caused by unmanaged automation changes occurred at a power plant operated by a PAS client, where a plant trip was triggered during the start-up of a Flue Gas Desulphurisation (FGD) system to treat flue gas from multiple generating units at the plant.
The units were at full load. The FGD supply fans tripped on high flue gas temperature, which caused the boilers to trip on high furnace pressure and this ultimately tripped all the units. Subsequent investigations determined that many environment projects were independently managed and placed into service without MOC of the automation systems. During the Selective Catalytic Reduction (SCR) installation, the boiler economizer surface had been removed to raise the inlet gas temperature to the SCR to 600 °F (315 °C). The resulting flue gas temperature entering the FGD flue gas plenum was 440 °F (227 °C). The FGD supply fans took suction from the plenum. The FGD supply fans tripped when the inlet gas temperature exceeded 375 °F (191 °C). That trip resulted in the furnace over-pressure that tripped all the units. This incident resulted in about $20 million in lost revenue.
As PAS’s survey revealed, automation changes occur so frequently that unlike other MOC scenarios, their sheer volume can quickly become overwhelming. PAS regards its integrity software as unique in importing the entire automation configuration database at regular intervals and comparing it with the database image taken at the last import. It then tracks and reports on all changes made between import intervals, both within a specific system and also among interoperable systems.
While most automation system manufacturers can track changes within their own systems, PAS software is designed to do so across system boundaries, and can therefore track signal genealogy throughout integrated systems. A common example of a signal crossing the boundaries of interoperable systems is a safety transmitter that is connected to an SIS, which passes the measured value to a DCS workstation for display, which in turn passes it to a historian for archiving. Integrity software provides this signal mapping and change tracking for more than 50 different automation systems.
Process manufacturers almost always standardise their underlying Microsoft Windows-based automation systems infrastructure by implementing a uniform configuration of hardware and software, often referred to as a Common Operating Environment (COE). Because most sites have a multitude of disparate automation systems, administrators accommodate the differences among them by defining multiple COEs within the facility and auditing them for compliance at regular intervals.
Managing modifications to this underlying automation system infrastructure is as important to safety and security as managing modifications to the actual configuration. It is commonly understood that the Stuxnet virus was initially introduced through a removable USB flash drive or ‘thumb drive’. An important consideration for process automation COEs is whether to disable the USB ports into which thumb drives may be inserted. Modifications to such things as the USB port enable/disable status or Windows active directory security settings represent a significant security concern and must therefore be under MOC control.
Building on the network scanning tools used by IT administrators, Integrity Recon from PAS tracks compliance to COE specifications for the servers, work stations and desktop computers on process automation networks including such items as:
- software installed on each machine and its versions and patches;
- system hardware enabled/disabled statuses;
- correct operation of active directories;
- compliance to available drive capacity guidelines;
- disabled and stale user accounts;
- application and operating system services status.
To ensure no unauthorised or malicious changes are lurking within your automation system, each change – whether to the actual configuration or the underlying infrastructure – must be under strict MOC control. Automation databases must also be checked continually for changes and compared with approved MOC cases to determine if those changes were actually authorized. Any changes that fail to reconcile would therefore be unapproved and require investigation.
PAS has introduced an intelligent management of change application called designed iMOC to address the special needs of automation. It provides robust MOC capabilities that are easy to engineer and use but, more importantly, it works with both Integrity software and Integrity Recon to reconcile the changes detected by those applications with the MOC cases that authorised them and ensure only approved and managed changes occur. Reports of unreconciled changes are automatically generated.
A streamlined MOC process
As most automation system configuration changes involve no changes to drawings or procedures, the traditional MOC process would be overkill. But to ensure that only safe and appropriate modifications are made to configurations, each change must be evaluated, approved and tracked. A shorter, more streamlined MOC process that entails no more effort than the modification itself is clearly needed for most parameter-only automation configuration modifications. In these cases, PAS suggests a special classification of the MOC process that is initiated, approved and closed out by the automation professional who actually makes the change. This process would employ a simple checklist form completed after the modification is made and electronically signed by the person making the change. It is assigned a tracking number and saved as an MOC case by the iMOC software and then – on the next import of the configuration database by the Integrity Software – the modifications are reconciled with the MOC case that authorised them (Figure 1).
|Figure 1: Reconciliation of modifications to a specific MOC case|
A report of all unreconciled modifications is generated so that unauthorised changes are immediately identified and communicated to the responsible parties. Unreconciled changes may then be accepted or a new MOC case may be created to authorise changing them back to an acceptable state. With minimal effort, this process ensures no unauthorised changes remain in the automation systems.
User-defined approval logic
Engineers spend a lot of time acquiring approvals to initiate cases, move them to the next phase, or close them out. New intelligent MOC software reduces the time spent gathering approvals by using Web 2.0 technology to ‘push’ them to the approvers at the appropriate point in the workflow for that change (Figure 2). User-defined approval logic may be applied so that, for example, if a primary approver is unavailable, a secondary approver automatically receives the approval request. And, if the approvers fail to respond in a timely manner, iMOC reminds them that they have an outstanding approval task. These features significantly reduce time spent on non-value-added activities associated with the MOC process.
|Figure 2: Using intelligent MOC software means workflows can be customised depending on the complexity of the change, and the approval points and responsibilities defined|
The PAS Intelligent MOC software also helps reduce engineering time spent on preparing initial Requests For Change (RFC) by automatically identifying and cross-referencing all configuration linkages and places where an automation parameter is used. This feature significantly speeds preparation of RFCs by reducing the time spent researching the scope of the change and also ensures that no critical use of the parameter under change is omitted from the process.
In conclusion, while MOC is a core process in process industry plants, the effort it requires for typical automation changes is so disproportionate that it is not always applied, leaving plants vulnerable to unapproved, undocumented or even malicious changes to automation systems. Such changes can only be eliminated as a cause of incidents through a rigorous MOC programme that reconciles automation changes to the MOC cases that authorised them. New software tools provide rigorous MOC for automation system changes with manageable MOC procedures and documentation so vulnerabilities can be identified.
Randy Cole is Director Technology Applications, Power, at PAS. Before joining PAS, Randy was Senior Consultant with the Electric Power Research Institute, Inc. (EPRI). He has more than 25 years’ experience in power plant operation, engineering and management. For more information on PAS please visit www.pas.com