An insider view of a cybersecurity training workshop for employees of Europe's transmission system operators.
After the hackers had stealthily accessed the SCADA system and blew the transformer with a loud bang, the defenceless employees had no option but to remove the control plugs and manually turn the machine back on.
"That's what they had to do in Ukraine," said Michael John, Director of Operations at the European Network for Cybersecurity (ENCS), referring to the world's first confirmed power plant hack: the 2015 attack in which about 30 Ukrainian substations were taken offline and roughly 230,000 people were left without power in wintry December weather.
"In a real scenario," John continued, "workers would then need to drive to each substation and repeat the procedure, perhaps in the dark or in bad weather, which could take hours."
"So we'd need to decide who gets power back first. Probably hospitals...?" said an employee.
Luckily, the hack described above was not part of a real scenario. Instead, it took place at a recent 'Red Team/Blue Team' (RTBT) training session for industrial control systems and smart grid cybersecurity, organized by ENCS for member companies of the European Network of Transmission System Operators for Electricity (ENTSO-E).
The members of Red Team (the hackers) and Blue Team (the defenders) were real employees with key responsibilities within Europe's power systems, as well as several managers who had come to observe. The group was a mix of job titles and skill levels, including SCADA engineers, IT systems administrators and two chief security officers. The setup mimicked a high- to medium-voltage substation using a major manufacturer's SCADA equipment, but John was quick to point out that for training purposes it "could be any" company's equipment and did not imply that this particular technology was any more or less vulnerable.
During the three-day workshop, the two teams switched off so that everyone would have a chance to both attack and defend a power network with typical levels of protection. For an observer, the most salient take-away was that no matter how good the cybersecurity defences in place, employees must also learn the skills to use them effectively.
Anjos Nijk, ENCS Managing Director, explains that those who attend the training have, on average, only 30 to 40 per cent of the skill level needed to successfully avert a determined cyberattack, he says, and the main goal of the training is "to build this knowledge".
The course also aims to get people with engineering histories and those with IT backgrounds talking to each other. "We have to bring them together," Nijk says, as both skillsets and kinds of knowledge will be needed in an attack scenario.
IT'S EASIER TO HACK THAN TO DEFEND
It's easier to hack a network than to defend it, says John, although hacking a network certainly isn't easy. The hackers "have to want it - it's not script-kiddie stuff," he says. Of the two major power system hacks that have succeeded to date - the Ukraine incident and the 2010 Stuxnet worm attack on Iran's nuclear programme - analysts suspect that governments or state-sponsored actors were behind them.
According to security experts, the Ukraine attack was probably the result of a 'spear-phishing' campaign that sent malware to employees of the regional DSO via email. Once an employee had opened the email, the malware allowed the hackers to steal login credentials and ultimately to shut down substations.
In the absence of such a time-consuming way in, Red Team in The Hague had to resort to quicker methods. For time purposes they were given a program called Metasploit, which is used by both hackers and security professionals who test system vulnerabilities.
Within Metasploit, Red Team could choose between a number of exploits (smaller programs that make use of specific vulnerabilities in the targeted system). By using exploits to stealthily access the server set up by Blue Team, Red Team commandeered a fictional employee's laptop and ultimately accessed the transformer controls - all in under two hours.
Meanwhile, Blue Team was also analyzing the system for vulnerabilities, outlining what would need to be done to fix the most urgent issues. In the interest of time, they were only allowed to fix seven of the many vulnerabilities they found on checking through the system. From weak passwords to unpatched software to access control errors to misconfigurations in SCADA servers, operator workstations, HMIs and network equipment such as firewalls and switches, the problems were not easy to locate and the fixes weren't quick to implement.
One Blue Team employee pointed out a potential vulnerability in the IEC 104 SCADA protocol, which connects to substations to receive information and allow control. However, when asked if he was comfortable that he could effectively analyze the protocol for vulnerabilities, the employee replied, "Not yet."
An unpatched database server that hadn't been updated in a while was identified as vulnerable, as were weak passwords, reachable from the enterprise network, which could "probably be hacked within minutes" according to John. But implementing such fixes after a hacker had already gained access wouldn't do any good. ENCS emphasizes that it's easier, much more effective and cheaper in the long run to mount a proactive defence.
In addition to making sure you understand the normal traffic on your network, ENCS says, a good defence must be proactive in a number of other areas. Much preventative work must be done: in procurement, to make sure your devices are secure; in system architecture, to minimize trust-based communications and mitigate the impact of a potential attack; in training, to raise awareness of vulnerabilities such as spear-phishing; in automated intrusion detection, in exhaustive systems testing and in formulating recovery plans for every possible scenario.
ENCS is working on standardizing cybersecurity requirements for its member utilities. But according to Nijk, the standardization process has presented a number of issues native to the energy sector. For example, the growing digitalization of power equipment and installations carries increased risk as more devices are connected, and this risk can grow quickly as digitalization becomes a necessity. But the power industry has a long heritage of thinking about risk in different terms - for example, insurance against a disaster that might, but is not certain to happen rather than costly extra fortifications - and organizations often move slowly to implement solutions, while hackers are agile "entrepreneurs by definition".
In terms of certification, Nijk says, the industry "won't do any more than is required" to meet the standard, as every action has a cost and companies are spend-averse - but "a hacker will think outside the box" to find vulnerabilities. Thus, any certification scheme needs to be exhaustive to ensure that not only the bare minimum is covered. And while every manufacturer wants their equipment to be the standard, in reality different technologies are going to be deployed, so the certification must be technology-agnostic and thus more complex to formulate.
ENCS sees its partnership with ENTSO-E as crucial to increasing cybersecurity for Europe's power sector. Although many new technologies that could potentially provide new vectors for hackers, such as smart meters or electric vehicle charging stations, operate mainly at the DSO level and thus policy and news headlines have tended to focus there, the group says we must not forget that the energy system is finely tuned and interconnected, and that DSOs and TSOs are interdependent partners. If a cyberattack compromises one, the other will suffer the effects.
ENTSO-E's insights into the European transmission system and the challenges faced by its operators bring "increased resilience and valuable wider conversations", says Nijk, adding that "as a membership organization, our strength is in the huge collective experience and knowledge spread across our member base. By collaborating with ENTSO-E, we gain visibility and insight from a wider and deeper pool of experts, and they can begin to benefit from the expertise we've developed."
Anjos Nijk, ENCS Managing Director, will be speaking at European Utility Week next month in Vienna. Click here for details.