By Ed Sullivan, Data Comm for Business, IL, USA
In the Supervisory Control and Data Acquisition (SCADA) environment today, communications networks are being forced off traditional analog networks onto lower cost, higher speed digital networks. SCADA host computers and Remote Terminal Units (RTUs) are also capable of higher speed operations. With more processor power, more computer memory and newer SCADA polling protocols increasing the data volume sent and collected from the RTU, the need for higher speed SCADA network operations is escalating.
Frame relay networking is one of the most commonly used methods of networking this higher speed operational demand, and it is relatively simple to use when supported with the proper devices.
Frame relay is a networking protocol for multiplexing data traffic over a Wide Area Network (WAN). Frame relay is most often used to reduce the costs of connecting to remote sites for several reasons. Leased lines are cost-sensitive to location, while frame relays are not.
Frame relay also supports a wide range of protocols via standard encapsulation techniques. It is also a synchronous HDLC protocol-based network in which data is sent in HDLC packets. These packets are referred to as “frames”, which are passed end-to-end through switches at a very fast rate. Frame relay devices within a frame relay network pass frames from node to node without error correction, and is therefore much faster than similar technology such as X.25 which performs data correction as packets are transmitted.
Digital lines connect the frame relay to the user premises. These lines typically range in speed from 56 Kbps to T-1 (1.544 Mbps) and faster. A Data Link Connection Identifier (DLCI) number identifies a virtual circuit endpoint. One or more DLCI numbers are assigned to each line endpoint. Over 900 DLCIs can be assigned, but frame relay networks rarely have so many because a user seldom needs more than one or two dozen. There may also be limits due to constraints in the frame relay switching equipment and how large a network can be realistically supported.
Frame relays also allow networks to be meshed. Since each endpoint can have one or more DLCI addresses, each user location can be logically connected to one other location, some locations or all locations. If every location were connected to every other location, the network is considered “fully meshed”.
Data Comm for Business (DCB) has been installing Frame Relay Access Devices (FRADs) for many years, and often tests frame relay networks for propagation delay in the real world. Today, round trip propagation delay from a host to remote and back has fallen below 50 ms on some installations, matching the propagation delays of modem-connected systems. In short, frame relay networks perform as well or better than modem-linked systems.
Mecklenburg Electric Cooperative is one of many rural cooperatives in the US that came into being in the late 1930s to supply electricity to customers too few and too remote for regular power companies to serve. The Cooperative has grown to 6436 km of line serving 27 948 Virginians in an area covering 4206 km2. Like many established utilities, Mecklenburg adopted a SCADA system for handling its telemetric needs.
Mecklenburg operates 22 substations that are managed remotely through radio modem links. Mecklenburg also utilizes a load management system that has saved the Cooperative a great deal of money since its inception in 1985. However, the hardware that supported the SCADA system was old and outdated.
“We were trying to keep our old system going as long as we could, and we managed to do that for a couple of years,” said Kim LaTulipe, electrical engineer with Mecklenburg. The SCADA system needed to be totally upgraded, but there was a problem involved. The upgrade would mean separating the SCADA system from the load management system that managed power during peak operating load times. Mecklenburg would have two systems that would be virtually incompatible with each other, and only one communications path. The choices to resolve the problem were limited.
Figure 1. Example of SCADA broadcast polling frame relay access device modem architecture
“We did not want to add a second communications path,” said LaTulipe. “The licensing of the frequency range we were using was frozen a couple of years ago by the Federal Communications Commission (FCC), so we couldn’t add an additional radio line. Leasing telephone lines was far too costly. Installation for the leased telephone line was estimated at $200 000 and the monthly costs of the leased lines would have amounted to $6000 per month for the 22 substations, or over $70 000 per year.
“Mecklenburg passes costs directly to our consumers, so we could not afford an expensive project. We had to find some way for the two systems to share the same communication path, in an application with a very low margin for error.”
One solution seemed possible: using statistical multiplexers (MUX) to run both the SCADA system and the load management system over the single radio line. “The engineer for the manufacturer of our SCADA system mentioned statistical multiplexers,” LaTulipe said. “They told me that they did not work well, and that they weren’t interested in helping us because they couldn’t guarantee that a statistical MUX setup would work.”
LaTulipe refused to give up and she started searching the internet for information on statistical MUXs. She soon discovered DCB and contacted Russ Straayer, DCB president, who assured her that they could install a suitable statistical MUX.
“Russ told me that there was no problem, they could help us use statistical MUXs to run both systems over the same radio line,” LaTulipe said. “All I was concerned with then was how much it was going to cost.”
LaTulipe found that it was going to cost only about a quarter of the installation charge to lease telephone lines, and there would be no continuing cost burden. Not only had LaTulipe found a technological solution to Mecklenburg’s problem of running two incompatible systems over the same communications path, but it was going to save the Cooperative over $70 000 in leased line costs per year. It seemed the ideal solution.
The overall project was fairly complicated and involved much more than just getting the statistical MUXs. “We were doing radio upgrades, RTU upgrades and additionally we were implementing a new communications system with the statistical MUXs. We had a lot going on at the same time and we were trying to make this all work together,” said LaTulipe. “Having one system work is pretty impressive, but getting three to work at the same time is a little hairy.”
Installation of the statistical MUXs was done in two phases: the one substation that did run on a leased line was to be installed first, with the remaining 21 substations scheduled for later. Basically, the existing multi-drop radial system would have the statistical MUXs added to it, creating two side-by-side systems that ran over the same communications path. This first project revealed one problem that was encountered with implementing DCB’s statistical MUXs.
Instead of sending data in packets, SCADA systems expect the data to be contiguous. Packeted data can generate communication errors that cause the system to not respond to polls. Unless the data can be properly transmitted and received, the system just will not work. LaTulipe encountered this problem with the first substation installation.
“DCB just did some programming changes on the chips in the MUX to make sure that the polls and responses were delivered intact. In the second phase of the project other problems were encountered with radio-keying error correction and data integrity,” LaTulipe said. “We had to make sure that the MUXs were set up to function in a half-duplex mode to key the radio modems properly. DCB also did some follow-up programming on the host end to disable error correction in the MUXs. This programming would also ensure the the request-to-send (RTS) on the remote end followed through to the host end, so the host would not think the remote was through talking before it really was.”
DCB had quickly resolved the problem. Without their help, LaTulipe doubts that the system would have worked and enabled Mecklenburg to save over $70 000 a year. “DCB provided us with a way to share the same communications path for two separate systems,” says LaTulipe.
The changes in power production and distributed systems demand that Utilities Plus, a municipally-owned energy services company, find new and better ways to capture real-time information from its members. To do that, and to help its 14-member utilities provide uninterrupted services at competitive rates, requires improving services through shared resources. That is the fundamental purpose of this not-for-profit organization. This four-year-old entity buys and sells a broad range of services for its municipal and public power utilities.
Figure 2. Typical SCADA network cloud set-up, using QEI computer system and Data Comm for Business equipment like that installed at Blue Earth Light & Water
Utilities Plus and its 14 area members needed to capture real-time meter and system status information so that operating decisions could be made quickly and cost-effectively. They decided that it was time for a load aggregation system. The vendors vying for this new system were quoting in the $250 000 to $300 000 price range, a cost that was considered expensive.
“That’s when we decided that our new, existing SCADA system could be used to aggregate the load,” says Paul Leland, the office manager at Blue Earth Light and Water. As one of the Utilities Plus members, Blue Earth Light and Water recently replaced its original Landis & Gyr network with a QEI 2000 frame relay SCADA system. The new QEI system uses a distributed 64-bit RISC workstation architecture to provide the maximum performance, flexibility, and scalability to meet Blue Earth Light and Water’s needs. The system was supplemented with a DCB Broadcast Polling FRAD.
Setting an example
“We decided to throw our hat into the ring and demonstrate to the other members of Utilities Plus that we could effectively use our new SCADA system to create a network cloud and aggregate the load,” Leland explains.
“Over the past six months we had been working on pieces of the system, including the communications side. We needed to develop some kind of wide-area communications scheme to meet our load aggregation requirements. We needed a low cost, easily maintainable, highly reliable communications product, and that’s where the DCB Broadcast Polling FRAD devices came in.”
QEI recommended the use of DCB Broadcast Polling FRADs, which work well with any SCADA system. Essentially, the DCB product is an async FRAD that allows multi-point polling over frame relay networks via RS-232 interface. Each master FRAD supports up to 160 RTUs, and is more cost effective than multi-point polling modem networks.
It only cost a few hundred of dollars for each master polling FRAD, so the DCB units helped to ensure that the communications cloud could be completed at a cost well below what vendors had been proposing earlier.
Cost-effectiveness was very attractive to Blue Earth Light and Water as well as the Utilities Plus organization. They had considered an internet-based communications alternative because it would have been practically cost-free. “We talked with several vendors who said they were already working on internet-ready RTUs and internet-ready meters, and so forth,” Leland explains. “But I don’t think we could rely confidently on an internet-based system yet.”
Leland says he was especially concerned with technological tradeoffs involved in integrating internet-based communications with SCADA systems that are continuously polling and relatively time-sensitive. “If we move to the point where we are monitoring load and coordinating the control of our generation assets as one logical entity, I just don’t think we can use the internet at that point. There are big issues with reliability and other timing issues with existing SCADA systems. Are we going to rely on the internet for that kind of control?
“What if an internet communications delay or failure jeopardized a power generation system at a critical point in time? The cost could be enormous in today’s energy market. I’m sure there will be a place for the internet communications for this type of application at some time in the future, but it’s probably going to be a few years from now.”
Another factor influencing the situation was a large investor owned utility, which sets the engineering benchmarks for power companies in the local region, provides standards on new generation equipment.
“They have all sorts of engineering specs,” Leland says. “And one of the current components is a communications tie between the new generation equipment and the IOU’s control centre. We have four or five cities that have fallen under these new engineering standards, and they are meeting the communications requirement via costly leased lines. Our new system will be able to eliminate those costs. Plus, once we aggregate all this data over our frame relay network back in Blue Earth, we will probably include the IOU in our frame cloud and just pass on all of the aggregated data to them from here in Blue Earth.”
So far Blue Earth has installed six Broadcast Polling FRADs and will be ordering another six to eight units as the other cities in the Utilities Plus organization get connected. Leland says the goal is to have that done by May 1, 2001.
“DCB is not just there to sell hardware,” Leland explains. “I got a good feeling from DCB. Now I would tend to use their service for any type of communication question in general. They seem to be quite innovative and yet personalized.”