HomeWorld RegionsEuropeMultimedia technology promises operator aids, improved plant efficiency

Multimedia technology promises operator aids, improved plant efficiency

Multimedia technology promises operator aids, improved plant efficiency

MOM technology is bringing plant operators more and better data for faster, more accurate decision making

By Dan Benson, Ph.D., Chellury Sastry, Ph.D., Siemens Corporate Research, Princeton/USA; and Dipl.-Ing. (FH) J?rgen Heilmann, Dipl.-Ing. Horst Hofmann, Siemens AG, Karlsruhe/Germany

Power plant operation and monitoring (O&M) systems will incorporate more and more multimedia information and technology in coming years, bringing a sense of visual reality into the control room and providing a more effective communication tool using a richer vocabulary of media.

Automated power plant control technology is used across a wide range of power production industries and, in many cases, the O&M system has become one of the most vital components in the power production process. As the level of automation increases, the role of the O&M system becomes more essential.

With the advent of modern O&M systems, the control room operator has been moved from the field to the control room and now relies on various sensor supplied data to manage today`s highly complex processes power plants. An operator`s effectiveness in decision making is dependent on his knowledge and awareness of the process at all times. It is therefore essential that a power plant O&M system present the operator with realistic information while providing means for effective communication.

It is well known that people learn and make effective decisions through association and reinforcement. Multimedia technology allows a set of data to be presented to a human via different mediums. It has been estimated that humans remember 20 percent of what they see, 30 percent of what they hear, 50 percent of what they see and hear, and 80 percent of what they see, hear and do simultaneously[1]. Effective use of multimedia information will result in more informed decision making, safer and more economical operation, and increased productivity.

The integration of multimedia information will play an important role in future power plant O&M systems. A great deal of work has already been done in the area of resource management for power plant control[2] and even interface design for advisory and diagnostic expert systems[3,4,5]. However, the addition of multimedia is of little use without proper multimedia data management and communication techniques to organize, retrieve, present, share and analyze this data[6].

Current O&M systems

Currently, most power plant operation and monitoring systems deal primarily with numeric data originating from process controllers, programmable logic controllers (PLC) and monitoring devices mounted out in the field. These values are often combined with alphanumeric and graphical data and presented on a display screen as an interface for monitoring, understanding and controlling the actual physical processes. However, text and graphics tell only part of the story and input sensors cannot measure everything. At times it is still necessary for operators to leave the control room to see what`s happening and/or they must rely on voice communication communicate with field technicians to relay verbal descriptions. Some applications make use of closed-circuit video monitoring of dangerous or restrictive areas but are primarily passive and separated from the control system.

As is typical in most applications, a great deal of the information regarding machinery, equipment, operating procedures, alarm situations and maintenance is found in hardcopy form. Manuals are voluminous, difficult to handle and non-integrated. Furthermore, specific information is often difficult to find within a manual. Valuable time is often wasted because information must be looked up manually. Finally, even after finding all of the cross-referenced pages within multiple manuals, the reader is then required to synthesize various information sources into a comprehensible picture.

One of the most important tasks is handling alarm situations in which an operator must be able to quickly comprehend the extent of the alarm, determine the cause and take recovery actions in a timely manner. Some systems have added diagnostic assistance with limited success, but operators generally rely on their memories or spend valuable time looking up needed information in supplemental reference manuals.

Future MOM systems

The use of multimedia information is often too easily regarded as a great improvement for many applications. However, because of the important role of the O&M system, multimedia technology will not be embraced by users unless it brings definite added value or benefit not provided by current systems. It is therefore important to introduce multimedia data along with advanced presentation, retrieval and management techniques that facilitate more intelligent use of the information.

With the introduction of any new technology, the eventual use and adaptation in the real world is rarely completely seen in the early stages of development. It is usually only after the new technology has been tried, tested and finally incorporated into an actual product that its true virtues are revealed. However, it is important to have an understanding of the purposefulness that multimedia technology will have in order to define the technology needed to achieve this vision.

Integrated hypermedia

The future MOM system will manage all information related to the process including on-line operation data, operating procedures, alarm handling, equipment layout and specification sheets, piping and instrumentation diagrams, archived trending data, and training and service maintenance. This multimedia information, coming from several different sources, will be integrated with the process control system, interconnected to form a hypermedia structure and available on line.

Information will come in many forms and be combined and organized in object-oriented structures that represent physical objects in the process, such as machines, equipment and materials. The physical objects of the process will be represented on the display in a manner familiar to the operator in terms of appearance, interconnections and behavior. Each object will have several different views with different levels of details, from which the operator can choose. For example, the operator may choose to view a burner from a live video source combined with all pertinent status information and indications of its relationship to other equipment. In addition, each object will have associated behavior, or actions that it can respond to. The operator can interact with and control these objects directly with their displayed representations.

With such a rich store of process control information, an operator will be able to view the overall process at various levels of detail depending on the current task. Graphical icons of equipment may be replaced with actual photos or even live camera feeds showing the processes in real time. Service technicians will have at their disposal detailed images and videos illustrating steps to be followed in certain maintenance procedures. From recorded videos, engineers will be able to monitor the operation of equipment and review the process to make needed changes and improvements. In addition, managers will be able to perform video trending of output over time that thus will helping them fine tune the overall process.

Future power plant facilities are already being designed using 3D CAD systems. The CAD models that are being generated include detailed graphics and physical specifications of all equipment, floorplans and layouts, as well as information about the equipment manufacturers, part numbers, etc. This information, which is already in computer-readable form, will be integrated with image and video data, making available all information related to the design, construction, operation and maintenance of the entire operation. Such a scenario would allow an operator to simply point on the display to a specific unit requiring servicing. When the service technician arrives, he would then have all of the service information, procedures, physical construction and history of operation about that particular unit immediately available.

Information retrieval and presentation

Operators of future O&M systems will be called upon to manage more complex operations and equipment. Quick access to needed information and comprehension of this information will therefore be vital to the success and maintenance of the operation. Several multimedia data management techniques will be used to enhance both access and presentation of multimedia process control information.

Depending on the current operation, the operator will be able to see the most important information needed at any time. Much of this information will be retrieved automatically by the system which knows about the current and upcoming tasks and remembers what information the operator used in the past. Of course, the operator will be able to control what information is retrieved and presented to customize the control display to suit his needs.

Different retrieval techniques will be used depending on the kind of information sought. For example, finding the physical location of a specific valve in a large plant will use spatial lookup methods to retrieve a floorplan diagram highlighting the valve`s location and an image of the valve showing its proximity to visual landmarks. Conversely, while viewing a photo or floorplan diagram of the plant, an operator will be able to select an area in the photo to retrieve information about the equipment found at that location.

As the use of image and video data to convey procedural information becomes commonplace, multimedia data management technology will include advanced methods for retrieving and browsing video based on content. With an underlying information structure, the system will know the current context of the operator`s actions such that a request by the operator for procedural information can be handled either automatically or, at the least, filtered down to a workable subset. For example, suppose an operator is in the middle of a complex setup procedure and wants to review a particularly delicate operation involving a furnace. The system recognizes from the operator`s previous actions the type of procedure and the equipment involved. Therefore, an appropriate query to the information base is formed automatically and the appropriate procedural videos and any other information are retrieved and presented.

Since it is not likely that every request will result in retrieving exactly what the operator wants, advanced video browsing techniques will allow the operator to find the specific information desired from a reduced set of videos retrieved[7]. Suppose in the previous example that five videos were retrieved instead of one. The operator would then be able to examine the five videos by quickly scanning their contents to determine the appropriate one.

Operators cannot be expected to have all pertinent information fresh in their minds at all times, particularly for infrequently performed procedures. In such cases, operators can benefit from a quick “refresher course” on the handling of a particular task or shutdown sequence for a particular piece of equipment. The system will anticipate the need for such a refresher course and have it queued automatically ahead of time to be viewed by the operator before the task is to be done. Again, taking advantage of visual data, operators will be able to view the information rapidly, leading to safer and more informed acts.

Visual data

In recent years, several applications have incorporated machine vision in the manufacturing process, such as inspection systems. We anticipate greater and greater use of visual sensors in future systems, from remote monitoring, to automated visual archiving, to visual controllers. This will be driven by application needs, made feasible through decreasing hardware costs, and made usable with advanced multimedia data management techniques. The addition of visual information, both live input and stored data, to power plant applications will lead to greater comprehension and understanding of the physical processes, more informed analysis and faster, more knowledgeable decision making.

Visual information will come from a variety of sources. Both stationary and mobile cameras will be employed to provide visual feedback from the field directly to the operator`s display. Camera operation will be automated by the system based on the current state of the operation and needs of the operator, or controlled manually by the operator, such as panning and zooming. Likewise, recording of camera input will be triggered automatically by some external event or performed manually by the operator for on-demand archiving. Recorded videos will be used for many tasks such as including analyzing processes in action in order to fine tune operations, comparing output over periods of time to ensure quality control, reviewing alarm situations to identify causes and determine preventive measures, monitoring worker activities and techniques and to verify that proper safety procedures are practiced and equipment inspection.

Visual input will not be limited to only the visible spectrum. Infrared, X-ray, and other imaging modalities will be used to capture otherwise “invisible” information. For instance, infrared can be used to reveal early stages of corrosion that is otherwise undetectable to the human eye and ultrasound imaging may find use as a non-invasive form of inspection for certain applications.

Alarm handling

In any process control operation, operators must respond quickly to alarm situations. This requires fast and accurate comprehension by the operator, which is directly related to the information presented and accessible at the time of the alarm. Most alarm situations happen infrequently, so in responding to alarms operators usually need to refer to supplementary information in order to determine the cause of the alarm and the procedures to follow for proper recovery. In addition to the cause of the alarm, operators also need to know the location of the alarm in order to know where to go, who to contact or what to do to prevent potential disasters.

The addition of multimedia data will enhance the operator`s alarm comprehension and response through advanced retrieval techniques and presentation of needed information when an alarm occurs. This will include live video as well as instant replays of important areas, location indicators showing exact locations of alarms and affected equipment, trend displays of controlled variables causing the alarms and establishment of communication channels with appropriate people utilized.

Cameras in the field recording specific areas will be triggered by the alarm to queue up the last few minutes of video that was recorded just before the alarm occurred. In this way, the operator will be able to see an instant replay of the events that led up to the alarm. In addition, live video feed from those same cameras will be available on the display so that the operator can see instantly the current situation of the most essential alarms.

Visual communication

Operators often need to make notes of things they observe. Future MOM systems will be capable of multimedia communication, such as visual note taking. Operators will be able to capture visually the current state of the process and send it to others with added comments. For instance, if the operator observes an unusual situation on his monitor, he may want to send a snapshot of it to the engineering department for consultation. On the captured snapshot the operator will add voice and mouse annotations to highlight portions of the image.

Technicians in the field or at remote sites will be able to communicate with the operator, as well as have access to information in the process control system using personal communication devices. Wireless voice and data transmission is already possible. Transmission of image and video multimedia at remote sites might be done by direct connection to the process control network or by means of short-range user-premises personal communication systems[8].

Live video can be fed back to the operator`s display screen by means of a remote camera mounted on the field technician`s hardhat. The field technician would also have a video-capable display device so that two-way video communication could take place between the control room and the field. For example, suppose a technician is about to perform some technical task. The operator will be able to retrieve the necessary images and video depicting the proper safety procedures to be followed. The operator also sees what the technician is doing and is able to warn of potentially unsafe acts by comparing what he observes on the live video with previously recorded and archived video clips. Videos showing proper procedures are then transferred to the technician on the site for review.


The use of the Internet for information access and exchange has been rapidly increasing over the past few years, particularly with the popularity of the World Wide Web (WWW). The most common means for Internet users to access and view web documents is through WWW browsers, such as the Netscape Navigator[9] or the Microsoft Internet Explorer[10]. These browsers, which are based on the international standard hypertext markup language (HTML), provide a consistent and standard means of describing document content independent of how the content is actually rendered. This makes it easier to produce, distribute, share and consume information and provides a great deal of flexibility to reach the widest possible audience.

One obvious usage of multimedia access over the Internet for the power plant industry is the delivery of electronic reference material. While paper-based media may be with us for a long time to come, the use of on-line manuals is rapidly increasing. Online access is not only more efficient and flexible but also much more economical in the long run in terms of production and maintenance costs. These benefits can be magnified when applied on an even larger scale by expanding the pool of users across a global network. A central repository not only means maintaining only a single pool of documents but also means that new information is immediately available and can be delivered and presented in the most appropriate means to meet individual customer needs.

Maintenance manuals, in particular, often describe physical tasks or make references to actual parts and machinery. Multimedia can add a great deal of realism to this information resulting in more precise and effective communication. The Internet and web technologies support a wide variety of multimedia types and more are being added every day. From a manufacturer`s perspective, the Internet and WWW provide an excellent means of delivering vital information to customers that can range from technical reference manuals, product catalogs and order status to customer service and technical hotline support. And with the ability to link different documents and sites, direct access to strategic partner sites and sites for third-party products can be easily cross-referenced, providing customers with a rich and valuable source of all their information needs. Additionally, the Internet can provide an efficient means of software distribution, from complete applications to upgrades, add-ons and individual software drivers.

While the Internet provides networked access to the global community, the so-called “Intranet” provides restricted access within an organization`s internal sub-net. Intranets are typically connected to the public Internet through a “firewall” that provides protected access from the outside. Such a net can have all the benefits of Internet technology but with secure access and the ability to add a great deal of customization. Until data communication over the public Internet is sufficiently secured, many businesses are choosing to restrict transmission of confidential information to their internal Intranets (Figure 1).

Recently, there has been an increasing trend toward deregulation of power plants. One possible result is that there may be several plants located at different sites serving different customer needs. The operation of these plants must be managed from the business headquarters. To this end, managers at the headquarters need to have available to them the relevant plant data so that they can optimize their business decisions and can study the impact their decisions have on plant performance and overall economic operation.

The Internet can also be used by a power plant to provide access to actual process data, whether it be archived data, real-time data or a combination of both. This might be done by simply introducing a web server application attached to either the control system itself, the database storing the process data or both. Certainly, a web browser interface to power plant processes is not intended to replace the main control system of the process but rather can serve as additional monitoring access with the ability of enhancing or customizing information retrieval and presentation, as well as delivery of selected process data.

With web-based access to process data, multimedia will play an important role in providing effective presentation, navigation and interaction of process dynamics. Depending on their purpose, web pages might range from presenting continuously updated information to presenting a snapshot of the process at the time the pages were accessed.

One type of web document might be classified as essentially static, in the sense that they are created once and their contents do not change. Another type of web document can be considered dynamic, that is, created on the fly as the result of a database query or other type of processing. Power plant web sites might also contain static as well as dynamically generated pages. This capability has tremendous potential in flexibility for providing customized views, varying levels of details and other specialized functions. Current web technology also makes it possible for web documents to be “active” by containing scripts, “applets” and other technologies that provide some degree of client-side processing. This has opened the door for delivering much more interesting and powerful multimedia documents. It off-loads much of the required computations from the server to the client, thus providing faster response for user input while reducing network traffic.

Time-based media, such as audio and video, are also becoming a feasible web media with so-called “streaming” technologies that allow these media to be rendered during network transmission rather than having to wait until the entire file is retrieved before being viewed. In addition to audio or video signals, one can think of streaming data from sensors that might be rendered as animated graphics on a web page. Client-side scripts in a power plant web page might provide communication between active objects within the page as well as accept and process input from the user, sending requests or commands back to the server for processing.

It is clear that a corporate power plant Intranet can provide a variety of data access and communication capabilities between company sites. In addition to electronic mail, or e-mail, for communication of text messages, more advanced multimedia technologies open up a variety of possibilities for more effective communication, particularly over the Internet. One such technology that is currently under development at Siemens Corporate Research is visual note taking through dynamic picture annotation, known as OPAL–Online Picture Annotation with e-mail. This system allows a user to add synchronized voice and mouse gestures/markings to a given picture. The resulting annotation is stored as a single message and sent via e-mail. Because it`s a dynamic annotation, the recipient simply plays back the annotation, hearing the sender`s own words as the markings are drawn exactly as they were recorded. The intent is to capture as much of a real-time conversation as possible with the convenience of delayed delivery. The OPAL message annotation composer window is shown in Figure 2.

Other multimedia data such as animation, audio and images used to illustrate maintenance, start-up and shut-down procedures could also be exchanged over the Internet. This kind of interaction is especially useful while troubleshooting in cases of component failures or other emergencies. Operators can establish setpoints in line with production goals and then monitor the process. In case of deviations, operators can then discuss these deviations with business executives by exchanging plant data over the Internet.

The future

Certainly, with the rapidly growing interest in multimedia, some aspects of these ideas are already being applied or are under development,[11, 12, 13, 14] but there remain some research and technical issues that need to be addressed in order to bring about all of the envisioned scenarios.

The integration of multimedia information will play an important role in future power plant O&M systems. Because of the essential role of the O&M system, multimedia technology will not be embraced by users unless it brings definite added value or benefit not provided by current systems. It is therefore important to introduce multimedia data along with advanced presentation, retrieval and management techniques that facilitate more intelligent use of the information.


[1] Multimedia, URL: https://nua.ie/nua/ DigitalAge/multimedia.html.

[2] J. L. Alty, et al, Managing Multimedia Resources in Process Control: Problems and Solutions, in Multimedia: Systems, Interactions and Applications, Springer-Verlag, Berlin, pp. 293-306, 1991.

[3] J. L. Alty, Multi-Media Interfaces in Process Control–A Methodological Approach, Proceedings of Systems, Man, and Cybernetics, pp. 361-366, 1993.

[4] K. Zinser and L. W. Herbst, Advanced process display and multimedia technologies for optimizing power plant control, ABB Review, 9/93, pp. 27-32, 1993.

[5] K. Zinser, Integrated Multi Media and Visualization Techniques for Process S&C, Proceedings of Systems, Man and Cybernetics, pp. 367-372, 1993.

[6] D. Benson, et al, Multimedia Data Management for Process Control, IEEE International Conference on Multimedia Computing and Systems, pp. 332-341, 1994.

[7] F. Arman, et al, Image Processing on Encoded Video Sequences, ACM Multimedia Systems Journal, Vol.1 No. 5, 1994.

[8] B. Z. Kobb, Personal Wireless, IEEE Spectrum, 20-25, June 1993.

[9] Netscape Communications Corporation, Netscape Navigator, URL: https:// netscape.com/.

[10] Microsoft Corporation, Internet Explorer, URL: https://microsoft.com/.

[11] Aloke Guha, et al, Controlling the Process with Distributed Multimedia, IEEE Multimedia, pp. 20-29, Summer 1995.

[12] K. Zinser and F. Frischenschlager, Multimedia`s Push Into Power, IEEE Spectrum, pp. 44-48, July 1994.

[13] D. Clark, et. al, What will a typical process plant look like in the year 2005?, URL: https://iac.honeywell.com/Pub/ Journal/a9504plant2005.html.

[14] K. Larson, Tomorrow`s Control Technology, Control magazine, pp. 24-30, August 1994.

Click here to enlarge image

The control room of the Voerde coal-fired power plant in Germany.

Click here to enlarge image

Previous articlePEI Volume 5 Issue 6
Next articlePEI Volume 6 Issue 7