NERC issues cyberattack lessons learned report

The electric reliability organization for the U.S. has issued a white paper describing lessons learned from what has been described as the first publicly disclosed disruptive cyberattack on the U.S. power grid. The incident -- and the report -- shed light on cybersecurity vulnerabilities and how companies can protect themselves from risk.

A September 2019 report by the North American Electric Reliability Corporation describes an incident on March 5, 2019 in which a cyberattack resulted in brief communications outages between the grid control center and several remote generation sites in the western U.S. According to the report, a flaw in the attacked utility's firewalls allowed "an unauthenticated attacker" to reboot them repeatedly, effectively breaking them. The firewalls served as "perimeter devices" -- devices connected directly to the internet, which serve as an outermost security layer, and regulate data traffic flowing between the generation sites and the utility's control center. Each time the devices rebooted, operators would lose communications contact with the generation for several minutes before regaining the link.

The report suggests that the hackers appear took advantage of a known flaw in the firewall's interface:

A vulnerability in the web interface of a vendor’s firewall was exploited, allowing an unauthenticated attacker to cause unexpected reboots of the devices. This resulted in a denial of service (DoS) condition at a low-impact control center and multiple remote low-impact generation sites. These unexpected reboots resulted in brief communications outages (i.e., less than five minutes) between field devices at sites and between the sites and the control center.

The NERC Lessons Learned report contains some key recommendations: update and patch all firewalls, and have a means of monitoring vendor firewall firmware releases and their implementation. These actions are key elements of a strong cybersecurity posture.

Protecting cyber systems -- whether for the control of electric generation or other business functions -- helps eliminate downtime, reduce business interruption, limit liability and reputational risks. Consider some of the following lessons learned from the March 5th cyber-attack provided by NERC:

• Follow good industry practices for vulnerability and patch management. Close monitoring of vendor firmware releases and their implementation is a key element of a strong cybersecurity posture. Firewall firmware updates need to be reviewed as quickly as possible after release for risk and applicability. Testing in a development (or “sandbox”) environment prior to deployment can test the patch’s potential to introduce new problems.
• Reduce and control your attack surface. Have as few internet facing devices as possible.
• Use virtual private networks.
• Use access control lists (ACLs) to filter inbound traffic prior to handling by the firewall; minimize the traffic through a denial by default configuration with whitelisting for the allowed and expected IP addresses. Limit outbound traffic similarly for information security purposes.
• Layer defenses. It is harder to penetrate a screening router, a virtual private network terminator, and a firewall in series than just a firewall (assuming the ACLs and other configurations are appropriate).
• Segment your network. Restrict lateral communication to necessary and expected traffic to reduce the impact of a breach.
• Know your exploitable vulnerabilities so you can pursue fixes. Maintain awareness of vulnerabilities and understanding of those in your environment through product vendor websites and user groups and third party resources, such as the National Vulnerability Database, SANS Internet Storm Center, Exploit Database, or others. Consider asking the Department of Homeland Security under the “National Cybersecurity Assessment and Technical Services (NCATS) program” (or a security vendor) to conduct external vulnerability scanning. Join the Electricity Information Sharing and Analysis Center (E-ISAC).
• Monitor your network. System performance monitoring increases the likelihood that brief communications outages with little actual impact to generator operations will be more closely investigated. This is how this lesson learned came to be. Use tools for firewall log analysis to detect events and support post-event investigations. This will provide information about the nature of attacks and exploits used. Report attacks and suspicious activity to the E-ISAC.
• Employ redundant solutions to provide resilience and on-line maintenance capabilities. Of the entity’s sites impacted by the firewall reboot, not all experienced communications disruptions. Following the event, it was discovered that the sites running firewalls in high-availability/redundant pair configuration maintained communications during the reboots. At sites utilizing this design, the secondary firewall maintained communications while the primary firewall rebooted. Firewall redundancy preserves functionality in the event of a single firewall failure. Firewall redundancy reduces impact of firmware updates since each firewall can be updated independently without interrupting communications during the update process.

 This post was co-authored by William Roberts of Preti Flaherty.