A cybersecurity training workshop for employees of Europe’s transmission system operators shows how far we have to go to be prepared, finds Tildy Bayar
After the hackers had stealthily accessed the SCADA system and blew the transformer with a loud bang, the defenceless employees had no option but to remove the control plugs and manually turn the machine back on.
“That’s what they had to do in Ukraine,” said Michael John, Director of Operations at the European Network for Cybersecurity (ENCS), referring to the world’s first confirmed power plant hack: the 2015 attack in which about
30 Ukrainian substations were taken offline and roughly 230,000 people were left without power in wintry December weather.
“In a real scenario,” John continued, “workers would then need to drive to each substation and repeat the procedure, perhaps in the dark or in bad weather, which could take hours.”
“So we’d need to decide who gets power back first. Probably hospitals…?” said an employee.
Luckily, the hack described above was not part of a real scenario. Instead, it took place at a recent ‘Red Team/Blue Team’ (RTBT) training session for industrial control systems and smart grid cybersecurity, organized by ENCS for member companies of the European Network of Transmission System Operators for Electricity (ENTSO-E).
The members of Red Team (the hackers) and Blue Team (the defenders) were real employees with key responsibilities within Europe’s power systems, as well as several managers who had come to observe. The group was a mix of job titles and skill levels, including SCADA engineers, IT systems administrators and two chief security officers. The setup mimicked a high- to medium-voltage substation using a major manufacturer’s SCADA equipment, but John was quick to point out that for training purposes it “could be any” company’s equipment and did not imply that this particular technology was any more or less vulnerable.
During the three-day workshop, the two teams switched off so that everyone would have a chance to both attack and defend a power network with typical levels of protection. For an observer, the most salient take-away was that no matter how good the cybersecurity defences in place, employees must also learn the skills to use them effectively.
Rene Marchal is a former chief security officer at Dutch grid operator TenneT who is now seconded to The Netherlands’ ministry of justice and security. He is also chair of ENTSO-E’s working group on critical systems protection. In the process of helping to organize the RTBT training, he said he had been surprised by the average IT employee’s lack of cybersecurity awareness.
“I was surprised at how much less [IT employees] relate to securing the network or IT process against malware attacks. I thought before that it was more integrated in their mindset,” he said.
“There is a really huge difference between keeping the system running and defending the system,” he added, “and it’s a little bit uncomfortable to think that your system is under-teched. As Churchill said, ‘War is too important to leave to the military’, and cybersecurity is too important to leave to the IT department.”
Marchal says the RTBT training is “not for cybersecurity experts, but for management to get a better understanding” of cybersecurity issues.
“Especially for higher management in key positions to make investments,” he says, “they should know what the risks are. In the old-school world it was less complex, and easier to overview the risks. Now we say ‘It’s so complex, leave it to the experts’ – cybersecurity professionals rather than everyday employees – but there’s a risk to that.
“Management is ignoring these risks a little bit, and is therefore not willing to spend the money. And because knowledge is so weak on digital processes, we are still dependent on experts,” he says. “So this course is a wakeup call to get little bit of awareness and learn about the vulnerabilities, but also to create commitment so people will go back to their home base and spread the news.
“Part of the coin in an efficient and hyperconnected world is that you also become more vulnerable, and if you don’t compensate it can be a nightmare.”
Anjos Nijk, ENCS Managing Director, agrees. Those who attend the training have, on average, only 30 to 40 per cent of the skill level needed to successfully avert a determined cyberattack, he says, and the main goal of the training is “to build this knowledge”.
The course also aims to get people with engineering histories and those with IT backgrounds talking to each other. “We have to bring them together,” Nijk says, as both skillsets and kinds of knowledge will be needed in an attack scenario.
It’s easier to hack than to defend
It’s easier to hack a network than to defend it, says John, although hacking a network certainly isn’t easy. The hackers “have to want it – it’s not script-kiddie stuff,” he says. Of the two major power system hacks that have succeeded to date – the Ukraine incident and the 2010 Stuxnet worm attack on Iran’s nuclear programme – analysts suspect that governments or state-sponsored actors were behind them.
According to security experts, the Ukraine attack was probably the result of a ‘spear-phishing’ campaign that sent malware to employees of the regional DSO via email. Once an employee had opened the email, the malware allowed the hackers to steal login credentials and ultimately to shut down substations.
In the absence of such a time-consuming way in, Red Team in The Hague had to resort to quicker methods. For time purposes they were given a program called Metasploit, which is used by both hackers and security professionals who test system vulnerabilities.
Within Metasploit, Red Team could choose between a number of exploits (smaller programs that make use of specific vulnerabilities in the targeted system). By using exploits to stealthily access the server set up by Blue Team, Red Team commandeered a fictional employee’s laptop and ultimately accessed the transformer controls – all in under two hours.
Meanwhile, Blue Team was also analyzing the system for vulnerabilities, outlining what would need to be done to fix the most urgent issues. In the interest of time, they were only allowed to fix seven of the many vulnerabilities they found on checking through the system. From weak passwords to unpatched software to access control errors to misconfigurations in SCADA servers, operator workstations, HMIs and network equipment such as firewalls and switches, the problems were not easy to locate and the fixes weren’t quick to implement.
One Blue Team employee pointed out a potential vulnerability in the IEC 104 SCADA protocol, which connects to substations to receive information and allow control. However, when asked if he was comfortable that he could effectively analyze the protocol for vulnerabilities, the employee replied, “Not yet.”
An unpatched database server that hadn’t been updated in a while was identified as vulnerable, as were weak passwords, reachable from the enterprise network, which could “probably be hacked within minutes” according to John. But implementing such fixes after a hacker had already gained access wouldn’t do any good. ENCS emphasizes that it’s easier, much more effective and cheaper in the long run to mount a proactive defence.
In addition to making sure you understand the normal traffic on your network, ENCS says, a good defence must be proactive in a number of other areas. Much preventative work must be done: in procurement, to make sure your devices are secure; in system architecture, to minimize trust-based communications and mitigate the impact of a potential attack; in training, to raise awareness of vulnerabilities such as spear-phishing; in automated intrusion detection, in exhaustive systems testing and in formulating recovery plans for every possible scenario. And the work gets more difficult when even the basic defences are often not in place.
“We don’t all have detection software installed to find strange things happening on the network,” says Marchal. “This is not implemented on a large scale.”
All of this comes with a cost, of course, and requires a degree of cooperation between IT and OT departments that has historically rarely existed. At TenneT, Marchal says, the two departments are “quite mixed, but in other companies they are different tribes.”
In security terms, there is “a big difference” between IT and OT, he says. “In the OT world it’s still possible to, let’s say, buy a car without brakes, with brakes just an option. It’s absolutely possible in the OT world for them to say ‘We don’t have cars with anti-lock brakes’. A small market failure, I will call it.”
And the lack of communication seen between departments on the ground extends to the highest levels. “The problem,” Marchal explains, “is that especially for the TSO landscape, the critical infrastructure conversation is nationally-driven rather than at the European level, so we receive different signals from the Dutch and European authorities about threats and measures to take. In the ENTSO-E landscape we are not able to communicate completely freely among ourselves, even to share indicators of compromised systems or strange patterns in the network.
“State actors are the most serious threat for TSOs,” he says, “so most important is that we have good cooperation with the national intelligence services.” He rates the UK’s National Grid very highly in terms of such communication. However, some other countries “think ‘We privatized our utilities and we completely shifted responsibility to the companies’, but companies don’t have intelligence services.
“Companies think in risk assessment, not in threat assessment. Governments are built to protect us against other nations; companies are built to deliver goods in a profitable way. With state actors or state-sponsored actors or terrorism, if a state attacks TenneT, it doesn’t really attack TenneT but The Netherlands or Germany, with TenneT as the vector – and then it’s another game.”
The need for standards
According to Marchal, because there are only a few vendors of big SCADA systems “it’s maybe confusing for them that we don’t have basic security requirements in Europe for all SCADA systems for the TSOs. This is not a good incentive for them to come up with such packages in basic configurations; right now they’re an add-on. You can blame the vendors, but also blame the buyers – are we able to create an interesting market for the vendors?”
ENCS is working on standardizing cybersecurity requirements for its member utilities, which number 19 so far, with more in the process of signing up. But according to Nijk, the standardization process has presented a number of issues native to the energy sector. For example, the growing digitalization of power equipment and installations carries increased risk as more devices are connected, and this risk can grow quickly as digitalization becomes a necessity. But the power industry has a long heritage of thinking about risk in different terms – for example, insurance against a disaster that might, but is not certain to happen rather than costly extra fortifications – and organizations often move slowly to implement solutions, while hackers are agile “entrepreneurs by definition”.
In terms of certification, Nijk says, the industry “won’t do any more than is required” to meet the standard, as every action has a cost and companies are spend-averse – but “a hacker will think outside the box” to find vulnerabilities. Thus, any certification scheme needs to be exhaustive to ensure that not only the bare minimum is covered. And while every manufacturer wants their equipment to be the standard, in reality different technologies are going to be deployed, so the certification must be technology-agnostic and thus more complex to formulate.
ENCS sees its partnership with ENTSO-E as crucial to increasing cybersecurity for Europe’s power sector. Although many new technologies that could potentially provide new vectors for hackers, such as smart meters or electric vehicle charging stations, operate mainly at the DSO level and thus policy and news headlines have tended to focus there, the group says we must not forget that the energy system is finely tuned and interconnected, and that DSOs and TSOs are interdependent partners. If a cyberattack compromises one, the other will suffer the effects.
ENTSO-E’s insights into the European transmission system and the challenges faced by its operators bring “increased resilience and valuable wider conversations”, says Nijk, adding that “as a membership organization, our strength is in the huge collective experience and knowledge spread across our member base. By collaborating with ENTSO-E, we gain visibility and insight from a wider and deeper pool of experts, and they can begin to benefit from the expertise we’ve developed.”
But this expertise can only go so far unless employees on the ground are trained and prepared to prevent or recover from a real cyberattack. One significant threat to this, Marchal says, is digitalization, which means a loss of resilience and “a single point of failure”.
“From a business continuity approach we go for the efficiency increases of digitalization, but there’s also a paradox: we’re so used to perfect working computers and data and electricity grids that we don’t even have fallback scenarios in mind,” he explains. “For example, we had an outage in The Netherlands a year ago and people were calling the police to report that ‘My iPad doesn’t work anymore because I can’t charge it’.”
One thing we tend to forget, he says, is that we “should accept that there is always risk in the world. It is not normal that we never have an outage.”
And he adds that “all TSOs are challenged by the regulators to do things as efficiently as possible. I can imagine a tension between modernization and making a system a little bit obsolete, from the regulator’s perspective, in order to have mechanical fallback scenarios. These are hard discussions for Boards to engage in, especially if they will not be compensated within the regulatory framework.
“We have the knowledge and the technology so that we can go back to mechanical steering of the grid if needed – although this is quite expensive. But if you go for full digitalization, keep the risk you’re taking in mind. You can still install mechanical machines on stations, but if you have no personnel with the knowledge of how to operate them, that’s a problem.”
And this, the human factor, is really the key. Back at the RTBT training day, Red Team’s hack would have caused a widespread blackout of at least 4.5 hours under real conditions. But today, instead of heading out into a snowy night to manually re-start their equipment, the employees break for lunch while analyzing the exercise: for Red Team, how the hack could have been accomplished more quickly and efficiently, and for Blue Team, the many things that went wrong in their defence.
With the skills they’ve gained, these power sector employees will undoubtedly be more prepared to deal with what cybersecurity experts say is the inevitability – “not if, but when” – of a major cyberattack. They will also be more ready to think of themselves, rather than their IT departments, as the first line of defence. But the employees gathered in The Hague still seem less than fully confident. For them, as well as for their TSO employers, the work has just begun.
Michael John is a speaker at Electrify Europe. To register, click here.