Recommended Practice for Patch Management of Control Systems

Patches for control systems, whether they are for functionality reasons or security concerns, must be guided by patch process management which identifies the periodicity and lifecycle. Having a comprehensive list of security-related patches using OT-specific patching tools for gathering entire vulnerability, software, and security patch information seems simple. However, Operation technology (OT) patch management is far from simple. 

A single solution does not exist that sufficiently addresses the patch management of both traditional IT data networks and control systems. Unexpected or sudden downtime of industrial control systems can have severe operational consequences. The Department of Homeland Security (DHS) Control Systems Security Program (CSSP) acknowledges that ICS operators or owners should have an integrated plan that helps identify a unique approach to patch management for control systems. 

In this article, we will identify issues regarding the patch management of control systems and recommend best practices to strengthen overall ICS security. Let’s get started with an introduction to Industrial Control Systems (ICS) and Patch Management.

Industrial Control System or ICS is a term that refers to different types of control systems and associated orchestration, such as systems, networks, devices, and controls used to operate and automate industrial processes. ICS operates differently based on the industries, and these are built to manage tasks electronically and efficiently. The devices and protocols utilized in industrial control systems are used in almost every sector and critical infrastructure, such as transportation, manufacturing, power plants, and water treatment industries. 

There are various types of Industrial Control Systems. Most of them are

• Distributed Control Systems (DCS)
• Supervisory Control and Data Acquisition (SCADA) systems
• Programmable Logic Controller (PLC)
• Remote Terminal Units (RTUs)
• Intelligent Electronic Devices (IEDs)

Patch management is the process of distributing and applying software updates. These patches are necessary to resolve errors, vulnerabilities, or bugs in software. Common areas that need patches or updates include applications, operating systems, and embedded systems. When an error or vulnerability is detected after a software release, a patch can be used to fix it. It helps ensure that assets within the IT environment are not susceptible to exploitation. Patch management help to maintain the security and health of control systems.

Patches for ICS, especially legacy systems, are applied either late or not at all. Legacy systems are sometimes not patched due to their proprietary nature, service age, perceived obsolescence, or because the patches are unavailable. ICS patches have conventionally addressed stability and functionality issues within the original code instead of enhanced security.

In the past, when a single ICS component failed, it could be easily detected, isolated, repaired, and shut down. Now, with the advent of enhanced network communication, a single ICS component vulnerability could lead to a cascading failure in the adjacent network system by enabling unintended, exploitable access. Detecting and isolating the cause of failure in the control systems becomes more difficult, leading to potential consequences. 

A few major problems between ICS and IT security need to be addressed while developing a cohesive plan for ICS patch management. These include

• Slower patch evolution
• Network integration of ICS
• Unmaintained and abandoned  hardware and software
• Differences in patch management
• Vulnerabilities disclosure
• Reliable patch information
• Embedded commercial off-the-shelf packages

To overcome these issues, there is a need to develop a cohesive patch management plan. 

Management policies are systemized as a plan that directs organization procedures. An effective patch management program includes the following components.

• Configuration Management Plan
• Patch Management Plan
• Backup/Archive Plan
• Incident Response Plan
• Disaster Recovery Plan

Let’s discuss these components in detail.

A configuration management program must consider these elements.

• Asset owners should maintain a functional software code library that contains the most recent and stable software versions used in industrial control systems. Moreover, controls must be implemented to prevent unauthorized access to operational code.
• The current hardware inventory of all ICS equipment must be maintained and available to authorized users.
• A current network schematic map that locates junction boxes, wiring, and connections for data communication needs to be maintained.
• Configuration documentation, including inventory lists and schematics, must be controlled to prevent public access. 
• The policies and procedures regarding configuration management plans are reviewed and updated regularly.
• Use Configuration Control Board to monitor, authorize, and control changes.

Consideration should be given to various components in the patch management plan. These include

• Exposure and vulnerability reviews need to be conducted by personnel knowledgeable of systems and their usage combined with those accountable for those systems. The person must have the authority to decide the urgency of patching activities.
• Urgency reviews need to be conducted to assess the operational risks and determine if rapid actions are required.
• Patch deployments or other updates to the system may cancel the ICS warranty based on the system and vendor. Issues must be resolved before patch deployment.

The asset owner needs to maintain a current and functional backup. This backup should be created and updated before any patching activities and provide a last healthy snapshot of the production system. The plan describes the

• Frequency of backup
• Functional requirements of creating the archive
• Backup retention period
• Backup verification process
• Physical storage requirements

Patch testing is essential in control systems due to requirements for high uptime. These recommendations should be implemented in patch testing.

• Testbed needs to be created simulating the operational environment closely and allows for software compatibility testing.
• Tests should be conducted to ensure that patch resolves all issues identified by the supporting organization.
• Tests should be conducted to validate that patch does not cause any conflict with existing applications.
• One or more test suites need to be developed that exercise the system functionality.
• Write procedures to verify that installed patches can be removed without affecting operations.
• Tests should be conducted to validate that the patched applications remain functional. 
• Procedures and checklists must be used for patching activities to ensure initial accuracy and repeated patching activities.
• Records of tests, patches, and configuration changes must be logged and documented in the configuration management record.

Actions defined in the response plan often initiate the patching procedure. Within the patch management context, various aspects of the incident response plan can be useful.

• Define a scheduled discovery process for identifying new vulnerabilities and their effect on ICS.
• Detect if patching and workarounds are available to remove vulnerabilities.
• Make a procedure to notify the Configuration Control Board to review the identified vulnerability and its operational impacts.
• Develop processes to report an incident or give feedback on issues detected in the patch process.

The disaster recovery plan is critical if the patch hinders the system functionality and can’t be removed successfully. Here are some recommendations for a disaster recovery plan.

• Have a testbed equipped with working hardware located offsite. 
• Organizations need to track the restoration backup time. It is also essential to verify that image restoration function and backup data work in real-world situations. It can be tested by leveraging the last restoration point to create the simulation test environment.

Vulnerability analysis related to patch management is a process of determining when and why a patch should be implemented in the control system. It is recommended that a patch review team should be utilized to assess and determine if an ICS is vulnerable to identified attacks. A method used to determine if an industrial control system is vulnerable to risks is called vulnerability footprint or attack surface.

The vulnerability footprint consists of four primary elements. These include

1. Deployment
2. Exposure
3. Impact
4. Simplicity

Here we determine how an asset owner can implement the patch process based on this scenario.

The asset owner has seven communication systems remotely tied to controls with only one server system using a vendor unique protocol. A web search for vulnerabilities regarding this protocol lists several websites. However, the references to US-CERT web pages that list the Vulnerability Notes information of NIST National Vulnerability Database CVE-id information will provide the most reliable data. US-CERT vulnerability notes define the problems, impacts, and solutions with the CVE number for a particular vulnerability. 

Deployment

A review of Common Vulnerability and Exposure (CVE) among several informational websites reveals that users of a certain protocol version are vulnerable. However, users of later versions are not susceptible. A quick investigation of the installed version of the protocol would determine immediately if further action is required. If the clean version is installed, all activities can stop as the vulnerability has been addressed. The older version is vulnerable, and the next step is to determine the deployment risk. 

Exposure

US-CERT vulnerability footprint review shows the exposure value as high, based on unauthorized access via the network. This data can be utilized to determine the ranking directly, or the asset owner can derive it after reading the report. It helps determine which vulnerability ranks high and needs immediate action as outside exposure means the control system is open to external internet access.

Impact

Assuming that the ability to modify setpoints or general controls does not exist on the system via the vulnerable protocol version, the impact is medium. Another concern regarding unauthorized access is that all linked network systems are vulnerable to cascading failure impacts from one system.

Simplicity

The final analysis is to determine what skills are required to exploit the vulnerability. The exploit required more advanced knowledge, so the ranking can be marked as low. US-CERT would recommend that the asset owner should deploy an updated version immediately. 

A documented process needs to be in place for monitoring new vulnerabilities and exploits. 

• Whenever a new vulnerability has been identified, the asset owner should determine if it impacts the control system in operation.
• If it affects one or more systems, an alternative method should be considered.
• If a workaround is detected, the patch should be assessed and scheduled as part of a regular patch cycle.
• If there is no workaround, the patch review team needs to analyze the associated risk.
• Factors that are considered in the evaluation include the main components of the vulnerability footprint discussed above.
• If there is a high risk, an immediate patch may be needed.
• Otherwise, if there are robust business constraints regarding patch implementation, it may be necessary to hold off the patching process.
• All applicable documentation and patch records must be updated once the patch has been implemented.

If the urgency determination needs prompt action and a workaround solution is not available or not an appropriate solution, then these actions should be taken.

• Create a backup and verify its integrity by configuring it on a standby system.
• Create a checklist for patch activities and configure the patch on the standby system.
• Test the patched system for operational functionality and compatibility with other coexisting applications.
• Swap the patched system into production and keep the unpatched production system as a standby for urgent patch regression.
• Monitor the patched production system closely for issues not detected during testing.
• Patch the standby system after the confidence is built with the production unit.
• Update software configuration management plan and relevant records.

The issue of prompt ICS security patching is a genuine cross-sector issue. There is no simple solution while implementing or evaluating patches on control systems. The first essential step in fixing these issues is to start open communications between IT security, production, process engineering, and senior management. In this article, we have identified the elements of an efficient patch management program, vulnerability analysis, and recommended practices that can be used to incorporate the patch management process to control systems and existing IT security plans.

• DHS Patch Management Process